A study finds a coding agent harness can add 3 times the cost

2 researchers at Shandong University, Liu and Han, compared 3 coding agent harnesses with 5 models. A harness is the control software around a model, with its system message, tools and memory control. The median result changed on 13 percent of tasks between 2 trials of one set-up. It changed on 13 percent between 2 harnesses. One task cost 3 times more with one harness than with a different harness.

How it works

The team used Claude Code, OpenCode and mini-SWE-agent with the same models. They used the 45 hardest tasks of SWE-bench Verified, a test of code repair, and 447 more tasks. All trials were offline. Some set-ups had 2 trials to measure noise.

The numbers

• On 447 tasks, Claude Code and mini-SWE-agent were less than 2 points apart.
• The system message and tool notes of Claude Code are about 16,600 tokens. The harness sends them with each step.
• This quantity sets the first cost. The number of steps multiplies it.

What is not known

OpenCode had a score 9 points less than the other 2 at most. Part of that difference comes from an output limit. The authors give no cause for the other part. The study uses one primary test, and the top closed model operated only in Claude Code.

Insight: A harness gain needs a new trial as proof

Google Research reports that one change raised the GLM-5 score by 11.5 points, but it gave no score from before the change. This study found that 2 trials of one set-up change by 6 points at most on SWE-bench Verified. The score race for harnesses seems to help the maker. The customer who tells the maker to give a new trial has the control.