A new checker cuts the score of one agent from 63.4% to 5.6%
A new test looks at the tools that agents used, and not only at the last answer. It shows that scores can be much too high.
Researchers did new scoring of 33,583 traces from 13 AI agents that use tools, for example calendar, weather, medical and retail tools. The method from before compares the last answer to a text. The new checker looks at the tool calls, their inputs and the results. On the same 306 traces, the score of Gemini-2.0-Flash decreased from 63.4% to 5.6%.
How it works
The checker uses set rules and no AI judge. It checks each tool call and each input. It also checks the condition after the calls, and if each value in the answer comes from a tool result.
The numbers
- One incorrect input that the tool accepts increased the error rate from 52.4% to 74.5% (7 models).
- A group of AI judges found only +2.5 points on the same pairs.
- Agents gave tool results with no tool call in 0% to 42.5% of traces, by model.
A rule-based monitor uses about 7 microseconds for each trace. Most of the decrease in incorrect answers came when the agent did not give an answer.
What is not known
The tests use tools that a simulator makes. The authors write that the harness frequently accepts a tool step and the last answer in one response. The source does not tell if other systems do the same.
Insight: The scorer decides the result of an agent test
ReplayBench tests also find that a tool result used again causes agent errors. Here, the checker finds +22.0 points of error from one incorrect input, and AI judges find +2.5. The person who makes the scorer sets the result. A vendor score from a weak scorer can show low risk when risk is high. Leaders must find out how a score was made.
Sources
Posted
