ResolveLab grades the steps of AI agents on a system
A small Qwen3 model taught on its errors got 7.5 points more than a copy taught on expert examples only.
A developer Ta0p1 published ResolveLab, an open-source test tool for AI agents that use software tools. A web server causes faults, for example a reply that did not get to the agent after a write. The tool grades the agent from the server records, not from the claim of the agent. On 120 locked test episodes, the DAgger method gave 7.5 points more.
How it works
The server causes faults such as rate limits and update conflicts. A teacher that persons wrote labels the states that the model got. This method is DAgger. The grader counts incorrect writes and two commits as failures.
The numbers
The increase is 7.5 points, with a 95% confidence range of 1.9 to 13.3. The 2 groups had equal safety violations. One loop caused 26 of the 33 failures of the expert only model and none of the DAgger failures.
What is not known
The servers are synthetic. The tests use 1 model family, Qwen3, 3 seeds and 1 DAgger step. Some agents in use have an expert to identify their states. The author says that AI helped to write the code.
Insight: Groups now grade the system, not the answer
ReplayBench found a mean error rate of 0.649 in the last system condition when agents used tool results again. A report of 12 October found hardware faults that cause errors that are not easy to see. Some groups at this time test the system after a fault, and the group with the tests that others use seems to set the test for a good agent. Make such tests for your tasks.
Sources
Posted
