What matters in AI.

Subscribe

A test of 22 harnesses finds that the top one changes with the model

In 27,280 agent runs, Codex CLI was in position 1 by cost for each task with GLM-5.3 and in position 16 with Kimi-K3.

The FrontierHarness leaderboard, made with Baseten, has a new version. It has results for 22 harnesses, which are the control software around a coding model, with five models on 62 terminal tasks. The authors tell us to think about a new harness when you change the model. The median harness moved 11 of 22 positions in cost rank between models.

How it works

Each harness did four tries on each task with each model. It used default adjustments and the maximum reasoning level. The top value is the minimum cost for each success.

The numbers

The model sets most of the cost: 92.8% of the range. With DeepSeek, Cline CLI had 5.0 times the cost of OpenCode v1 for 2.0 more points. Audits found 216 trials of reward hacking, which is an incorrect result on a test. Each of these trials gets a score of 0.

What is not known

Each cell has one value, and the effect of the pair also has noise. Codex CLI with GLM-5.3 was the top harness in 99.6% of 4,000 redraws. The other four top harnesses are only the most possible of some close harnesses.

Insight: The harness choice hides a cost lever

In October 2026, researchers at Shandong University found that one task cost 3 times more with one harness than with a different one. The CUGA team, which tested its harness, tells that one model had 30% to 81% with three harnesses. This test adds that the top harness changes with the model. It reads as a control that most teams do not see: the default harness can set your cost. Test the pair.

Sources

  1. In 27,280 coding agent runs, 64% of inference spend went to runs that failedrunta.com
AI MATTER · NEWS · AI MATTER · NEWS ·17 NOV2026

Posted

Tags

Learn the terms in this story

More in Research

All Research news