A clean history can make an AI agent worse when the error remains

Researchers tested a system that compacts, repairs or hands off the history of an agent after an error. They froze agents in the middle of a task. They changed the history or the world, and a second model continued. A history that looks clean, while the world has the error, can make an agent worse than no change.

How it works

On AgentCE-Bench, an agent writes an incorrect value. The team adds the correct value as a message adjacent to the error, or as a completed step of the agent. On AppWorld, a database write gets a receipt that says success or defect.

The numbers

• A correct message raised recovery from 51% to 66%.
• The completed step decreased recovery to less than the unchanged history.
• A success receipt cut task success from 45% to 28% when the effect was missing.

What is not known

The two effects have a distance of 27 points on two receiver models. The tests use two benchmarks. The paper does not tell how far the result is correct in other systems.

Insight: Agent history is a part that you must test

Earlier tests found that memory faults cut the correct answers of agents by 40.54 points. They also found that removal of the history of thought steps moved scores from minus 4.4 to plus 51.3 points, for each model. This paper adds that a clean history can cause damage. Memory tools select the facts that an agent believes about earlier steps, and they seem as important as the model. Test the history layer with the model.