Of 1,403 AI agent errors, 725 recover without a person
The authors say that persons must work first on the errors from which an agent cannot recover.
Researchers made ReInBench, a dataset of 1,259 agent traces (records of each step) with 1,403 errors from computer-use and web tasks. They find that 725 of the errors recover, frequently in a small number of steps. They also made ReCoF, a method that predicts if an error will recover and thus tells a system if a person must work on it.
How it works
The authors study how frequently and how quickly agents recover. When an agent recovers, it can also recover from subsequent errors of the same type, and it can finish the task. The rate is different for each model that operates the agent.
The numbers
ReCoF increases the Macro-F1 score (a score of how correct a method is) by 9.02% on average. This is a ratio to the top other method in the test, not an increase in points.
What is not known
The text gives no rate for each model and no cost of the method. The data includes only computer-use and web tasks.
Insight: The hard part is to tell which errors recover
Most errors in this data recover, and thus a stop at each error is frequently not necessary. A different study of 8 models found a final-state error rate of 0.649 when a tool result is used again for a different task. The monitor that tells the 2 types of error apart reads as the tool that sets when a person must work.
Sources
Posted
