All 16 changed AI agents cannot do some tasks they did before

Researchers study AI agents that change the harness of the agent, the control software around a model, to get more points on tests. The model weights, the numbers in the model, stay the same. In tests of 4 methods on 2 task types, all 16 changed agents get new tasks and drop tasks they did before.

How it works

Harness evolution lets an agent get more points with no change to the model weights. The team tests each method again and again. It removes one part at a time to find the work of each part. A follow-up method, EvoAudit, changes 4 steps of the method.

The numbers

• Reliability goes up 5.2 points in 1 test, but the agent drops 10 tasks.
• Most reliability increases also increase the deployment cost.
• The agent with the top score gives the most increase in none of 4 pools.

What is not known

The increases do not move in full to new tasks. Coverage usually decreases on the general-task target, but it increases on the coding target with one harness type. The paper gives no score for EvoAudit.

Insight: One score does not show what an agent lost

Google Research writes that one harness change gives GLM-5 a score of 11.5 points more. It gives no score from before the change. This paper finds that all 16 changed agents drop tasks. Tool makers show one score, and that score reads as a view without the drops. The buyer must test the tasks of the buyer. Test your tasks again after each change.