What matters in AI.

Subscribe

A mean pass rate does not show the defects of an agent test suite

For 4 of 5 traceable defects, the most that a defect can change the suite score is at or below the 0.10 detection threshold.

Researchers examine a procedure to make agent tests from the specification of an agent. A generator fills a space of requirements, personas (types of user) and scenario types, and a mean pass rate scores the suite that it makes. The authors put 7 controlled defects in a tau2 bench agent, and report that this score does not use the results of the search.

How it works

If a defect touches b scenarios that give a correct result at baseline, it can add at most b/N to a suite of N scenarios. The authors report that a score for each cell, which is one mix of requirement, persona and scenario type, finds more than the mean.

The numbers

  • A defect that removed a tool made all the scenarios that it touched incorrect.
  • The suite score moved in the opposite direction.
  • A score for each cell found that defect in full.
  • A suite of only happy-path scenarios (easy tasks) found no defect. A suite of many types found one defect in 5 of 20 draws.

What is not known

A forced escalation defect was found in retail at a traced coverage of 0. These defects show in behavior and not in the recorded steps. The paper does not say if the results hold for other agents.

Insight: A pass rate alone is weak proof for an agent

Arize shows a test result that went from 5 of 16 to 48 of 48 reports after a change. This paper shows that a suite score can move in the opposite direction when a tool is removed from an agent. This reads as a move to make coverage the standard for agent tests. Tell vendors of agent tests to give coverage.

Sources

  1. Generated Evaluation Suites as Diversity-Driven Search for Real Agent Regressionsarxiv.org
AI MATTER · NEWS · AI MATTER · NEWS ·16 NOV2026

Posted

Tags

More in Benchmarks

All Benchmarks news