A model that finds agent faults well can monitor agents badly
GPT-6.1 Sol is one of the best at diagnosis of failed agent runs. As a live supervisor it was the weakest, at minus 14.5 points.
Researchers released ReplayRCA. It tests if one AI model can find the main mistake in a failed agent run and repair it with one step. The score is the success rate after the repair, minus the rate of a placebo that replays the first step. Of 13 frontier models, the best, Grok 4.7, got plus 51.3 points on 62 failures.
How it works
The test restores the task to the condition before the chosen step. It replaces the step of the agent with the proposed step, and the same agent continues. One track shows the full failed run. In the other track, a live supervisor monitors the run and can act one time, before a step that nobody can reverse.
The numbers
- The top eight models are statistically the same.
- The flagship models of the two largest vendors rank 8th and 12th.
- A strong model that must act at the last step loses 17.1 points.
The two best live supervisors are the same, and one is a small open weight model, DeepSeek V4.1 Flash.
What is not known
No model repaired 5 items. The live track used 87 runs. All tasks come from one generated office environment, OpsWorld.
Insight: Skill at diagnosis does not give skill at live oversight
A method of 9 October finds the step that starts an agent failure, and a debugger of 17 October lets an agent replay a fault. Those tools work after the fact. This test shows that skill after the fact does not transfer to live oversight. A buyer who picks a monitor on a diagnosis score can pick the wrong one. Test monitors live, and try small models.
Sources
Posted
