Agents that do a task recover from errors in only 47% of tests

Tanmay Sah and Dolly Sah gave 2 results, UndoBench and EvoUndo, on how agents that write code recover from errors. In 2,880 tests, agents did 84% of the tasks with no made error. When the team made an error, the agent could recover in only 47% of the tasks that were correct before. An agent can do a task and still not put the system back.

How it works

UndoBench does each test 2 times, with no error and with a made error. A timeout shows that no reply came, not that the step had an error.

The agent can do the step again and make 2 tickets. EvoUndo keeps the initial condition before it changes an agent. A different tool then checks that the undo put it back.

The numbers

• A step done again made the effect 2 times in 54% of tests.
• Of 600 changes to agents, 478 gave more points. In 197 of these, recovery was not correct.
• The first recovery method repaired 0 of the 197. A method of more steps repaired 180.

What is not known

The authors made this work, and the text shows no external check. It does not tell which agents are in the tests.

Insight: More points in tests can hide a recovery problem

In November 2026, a study found that all 16 changed agents could not do tasks they did before. This work adds that 197 of 478 changes with more points had bad recovery. A change that adds points and removes recovery can go through a score test, until a new check tests recovery. Teams that add this check keep the increase and prevent the problem.