A mean pass rate does not show the defects of an agent test suite
For 4 of 5 traceable defects, the most that a defect can change the suite score is at or below the 0.10 detection threshold.
Researchers examine a procedure to make agent tests from the specification of an agent. A generator fills a space of requirements, personas (types of user) and scenario types, and a mean pass rate scores the suite that it makes. The authors put 7 controlled defects in a tau2 bench agent, and report that this score does not use the results of the search.
How it works
If a defect touches b scenarios that give a correct result at baseline, it can add at most b/N to a suite of N scenarios. The authors report that a score for each cell, which is one mix of requirement, persona and scenario type, finds more than the mean.
The numbers
- A defect that removed a tool made all the scenarios that it touched incorrect.
- The suite score moved in the opposite direction.
- A score for each cell found that defect in full.
- A suite of only happy-path scenarios (easy tasks) found no defect. A suite of many types found one defect in 5 of 20 draws.
What is not known
A forced escalation defect was found in retail at a traced coverage of 0. These defects show in behavior and not in the recorded steps. The paper does not say if the results hold for other agents.
Insight: A pass rate alone is weak proof for an agent
Arize shows a test result that went from 5 of 16 to 48 of 48 reports after a change. This paper shows that a suite score can move in the opposite direction when a tool is removed from an agent. This reads as a move to make coverage the standard for agent tests. Tell vendors of agent tests to give coverage.
Sources
Posted