A 120B AI model did no better than a 20B model with tool faults

Researchers made MCP-Injector, a tool that causes faults in the requests of an AI agent to external tools. The requests use the Model Context Protocol, a standard connection between agents and tools. In 1,350 test runs, the task score decreased 60%. A model with 120B parameters did no better than a model with 20B. They made the tool to test agents when tools stop working.

How it works

MCP-Injector is between the agent and its tools. It can make a request stop working for good, one time, or from time to time. The fault mix comes from 57,240 issue and pull-request threads in 1,774 open-source projects for the protocol.

The numbers

The test used 2 models, 3 agent stacks and 3 fault conditions. The 120B model kept no larger share of its fault-free score. Its completion rate with faults was 91.1%, against 95.6% for the 20B model.

What is not known

The test used only 2 gpt-oss models. The authors give a 95% range of -0.2 to 9.3 points for the difference in completion rate. They write that error handling in the framework looks to have more effect than the scale of the model.

Insight: Agent reliability looks to go from the model to the code around it

This test found that a larger model did not help with tool faults. Other tests this month found a mean error rate of 0.649 across 8 models. This was when an agent used a tool result again. A different test cut the fault selection rate from 86.2% to 2.9%. This reads as control with the teams that make the error handling around the model. Find out who has that layer in your stack.