AI agents remove about 1 third of task rules when they compact

Researchers did a test of context compaction. This step frees space in the memory of an AI agent in a long task. They used 5 methods in the OpenCode agent with 3 frontier models. The step can silently remove task rules. The method and the threshold had less effect than the rules that the compaction removed.

How it works

When a task is longer than the context window, the agent must summarize its work. This step always loses some information. The team used PaperBench, ProgramBench and PostTrainBench.

The numbers

For the 2 models with 1M context, the threshold did not change the score. No method was better than the default self-summary. Some methods decreased the score of some models by a maximum of 20 points. In sessions of users, rules on how to work were removed about 6 times more than specifications of the thing to make.

What is not known

The authors write that benchmarks are friendly to compaction, because the goal is fixed and the work stays on disk. A task where the goal changes can lose more. The source does not test fixes.

Insight: The compaction setting is not the risk, the removed rules are

A developer set agents to compact at 42% full to keep space in a decompile task of 1 month. This paper finds that the threshold had little effect on the 2 large models. But rules that are in no other part are removed. Teams set compaction to save space, and the primary risk looks like the hidden loss of rules. Check which rules your agent removes.