Microsoft releases Scope, an open source test tool for coding agents
Scope operates one task with different coding agents and scores each result, but you must operate it on your own systems.
Microsoft released Scope as open source with the MIT license. It operates the same task with agents such as GitHub Copilot and Claude Code, and it scores what each agent makes. Microsoft calls the field Developer Agent Experience (DevAx). The point is to find where an agent has problems with an SDK (software development kit), a document or a tool.
How it works
You give Scope a task, a start codebase and an agent setup. Success is a set of criteria in a dependency graph, where one check can gate a different check. A judge tool, based on an LLM (large language model), scores each test.
Each test keeps logs and tool calls, and you can find a check with a bad result. You can change the model, do the task again and compare.
The proof
The source gives no test results. Microsoft says the judge can make mistakes and that important results must have human inspection. Results change from test to test. Do each task again.
The catch
Scope has no hosted version. A local setup must have Docker with Compose v2, Node 22, pnpm, mkcert and the GitHub CLI. The default worker must have a GitHub Copilot entitlement, and the judge must have credentials for a model provider.
Insight: The test helps decide which tools agents select
Scope can test if an agent selects your SDK at all. OutSystems opened Agent Experience to coding agents, and Aveni has two tools that test and monitor agents. This reads as a new contest for the tools that agents select, and the maker of the test helps set the rules. Builders who publish an SDK or an MCP server can test before agents select for their users.
Sources
Posted
