Same model got 30% to 81% on tasks with three different harnesses

The team behind CUGA is part of IBM. CUGA is an open-source agent harness. The team tested three harnesses with one open-weight model, GLM-5.3-flash, on 90 AppWorld tasks. A harness is the control software around a model that lets it use tools. Task results moved from 30% to 81% when only the harness changed. CUGA used a setup that the other two did not.

How it works

CUGA moves business rules out of the prompt and into the harness. Approvals and tool guards stop the use of a tool, and the model cannot change that. In a demo, an agent without rules paid a denied claim. The agent with rules held the payment for approval.

The numbers

The test has 90 tasks in 30 scenarios. A scenario passes only when all three of its variants pass. CUGA passed 20 scenarios. Hermes passed 8 and Deep Agents passed 2. The runs used no rules, and they do not test rule enforcement.

What is not known

This is one sample on one benchmark, and the setups were not the same. AppWorld uses consumer apps, and it is only a proxy for business work. The authors say Hermes and Deep Agents were built for other work. They tell others to send other setups.

Insight: The harness sets both the score and the risk

Here the harness moved the score from 30% to 81%. Other work shows that a harness can give a frozen model 28.7 points more than a different harness. A science harness made GPT models obey more dangerous chemistry requests. This reads as a race to control the harness, and the harness sets results and risk. Buyers must test the full agent, not only the model.