Test of 2,000 agent runs finds context compression gains are small

A team tested a small proxy that makes large tool outputs shorter before an AI model reads them. It sits between an agent and its MCP servers, which give the agent its tools. In more than 2,000 logged runs, the savings for full tasks almost disappeared. The team used local open-weight models, mainly Qwen3 4B, and short tasks, and it calls this one corner of the field.

How it works

The proxy cuts a large output and keeps the full output one call away. The model frequently needs the part that the proxy cut. It then tells the proxy to send the full output, and the team pays two times. A search mode that sends only the lines needed fixed this. The cost of 17% more went away, and on servers in use the proxy cost the same as no proxy.

The numbers

• Bulky synthetic tasks: savings of about a sixth.
• Plain truncation: a quarter of tokens saved, but the answer was not found in about one task in six.
• A bug in the proxy: 0 of 16 tasks correct, against 16 of 16 without the proxy.

What is not known

The tasks were short and the models were small. The team says its sample for the tool test is small. It did not test large models.

Insight: Vendors sell the cut in length, not the saving

Kong says its Code Mode gives an agent no tool definition for each operation. This test found that one more tool alone made a 4B model worse. This reads as vendors selling the easy number, the cut in length, while the cost is in full tasks. The buyer must set the test. Tell each vendor to give the tokens of full tasks and the share of answers not found.