Parts with hidden test names were defective in 80.6% of agent runs

Researchers audited benchmarks that grade coding agents with tests from the upstream project, the first software. The task text does not tell the names that the tests use, for example a module path. The authors name this a specification-oracle gap. On SWE-Milestone, software parts with this hidden requirement were defective in 80.6% of runs, against 48.4% for comparable parts.

How it works

The team made a static audit, which reads code and does not operate it. It covers five languages and finds each name, path and literal value that a test uses. Its held-out precision is 89.2% and its recall is 92.9%.

The numbers

• In 98 milestones and 35 published runs, 25 milestones have a test with a hidden requirement.
• When one hidden name is removed, the tests could not compile in 126 of 128 tests.
• In four more benchmarks, a third or more of the tasks are affected. Their specifications are release notes or LLM summaries.

What is not known

The leaderboard ranking stays the same when the affected tests are removed. The cost of a gap changes with how a harness grades. The authors release the audit and the fixes.

Insight: A benchmark can grade the copy of names, not skill

Microsoft released Scope, which scores coding agents on tasks that teams select. This paper shows that a task can be unfair when its tests use names that the text does not give. The game seems to be that the person who writes the grader sets the winner. Agents that copy upstream names get more points. Teams that make their own agent tests must audit them first.