Deleting AI reasoning history shifts task scores by up to 51 points

Researchers measured what happens when an AI reasoning model loses its earlier reasoning notes during a session with many turns. They tested 7 open-weight models, which are models with public weights, on 794 tasks. The same deletion changed scores by between minus 4.4 and plus 51.3 points, depending on the model. The authors say that this effect belongs to the model and not to the setup.

How it works

A reasoning model writes a reasoning trace, the written steps of its thinking, at each turn. Later turns read this history. Providers tie each trace to the company that made it. A request that goes to a different provider arrives with the history deleted. Before this paper, nobody had measured the task cost.

The numbers

• The test used 7 models, 3 seeds, 794 tasks and the official checker.
• The effect ranged from minus 4.4 to plus 51.3 points.
• For one model, the reading varied by at most 1.5 points from 0 to 100 percent deletion.

What is not known

The text names no hosted model. The authors say that a retrained model is a new model. They propose a two run screen, one run with history and one run without, to read a model again.

Insight: Switching AI providers can carry a hidden cost

Earlier tests found that a change in the order of choices changed up to 72.6% of the answers of one model. This paper adds that deleted history costs each model a different amount. Each model has its own hidden weak points. Software that moves requests between providers to save money seems able to lose accuracy that no bill shows. Measure each model before you route to it.