Check finds faults in 30% of the Terminal-Bench test for AI agents

Researchers examined Terminal-Bench 2.0, a test that gives scores to AI agents that work in a terminal. They found faults in about 30% of the test. They released the repairs as Terminal-Bench 2.1. The authors tell that accuracy increased by more than 10 percentage points. They also tell that leaderboard positions moved by a maximum of 3 ranks.

How it works

The authors made a workflow that checks a test again and again after its release. It uses reports from the community, task files, agent traces (logs of agent steps), automatic checks and approval by a person for each repair. Models can find correct results that the checks before the repairs reject.

The numbers

The full ranking stays near the ranking before the repairs. An increase of the time limit also changes results. It closed a 20 point difference between 2 agent setups on GPT 5.4. The authors name the bigger difference between score and skill “benchmark maintenance debt”.

What is not known

The paper examines one test. It does not tell how many other tests have the same faults. The authors tell that task checks and time limits must be checked again when models change.

Insight: A benchmark score depends on who sets the test rules

Terminal-Bench had faults in about 30% of its tasks. An increase of the time limit closed a 20 point difference. In October 2026, AOS-Bench showed 2 systems 0.6 points apart on single tasks. As a stream, they completed 54% and 68% of the tasks. A score follows the rules of the test, and the company that sets time limits and task flow can move a ranking. Test vendor statements.