A small task change cuts one coding model score from 74% to 53%

The authors of MaxxBench, a new test, changed 89 tasks from DeepSWE, a coding benchmark (a standard test). They changed the instructions or the code at the start. Claude Opus 5.5 and Gemini 3.8 Flash each get about 74% on DeepSWE. On MaxxBench they get 76.4% and 53.4%. The authors tell that many models guess the result that the hidden tests want. They do not follow the new instruction.

How it works

The authors made a small change to each task. In one task, the names of the report documents must be in lowercase. The correct code has a median of 17 more lines.

The numbers

Across 12 setups of models, the difference between the 2 end scores goes from 20.9 points to 48.3 points. About a quarter of the errors of 3 models occur when the model knows the new instruction and continues the first behavior.

What is not known

The authors made the test and give the results. The source gives no test by a different group. The authors tell that a benchmark is not used up while small changes to its tasks can show a difference between models.

Insight: A public score is a target for makers and attackers

The authors tell that a benchmark that many teams use goes into training data. A report on BenchDoor showed that training data made to increase a benchmark score also added a hidden backdoor in all 30 setups. A public score is a target for all teams and for attackers, and it reads as a weak measure of quality. Leaders must not select a model on a public score only.