Small AI models play an incorrect move again after they see it

A researcher did a test of eight small models in three games: Sokoban, Connect Four and Othello. The control software put the incorrect move in the record and told the model again. This increased the risk of that move. The instruction “do not do it again” did not help. With greedy decoding, 38% of the moves that the model was about to play came again.

How it works

Agent software can put an incorrect move and its error message in the transcript. Then it tells the model again. A move can break the rules, or it can follow the rules and give a bad result.

The numbers

The models contain 135M to 1.7B parameters. The effect is strong for moves that break the rules. It is not strong for moves that give a bad result, and not all models show it. A clean new command with no record is a bad method. A ban on the same move stops copies, but not moves that break the rules.

What is not known

The test uses only small models. A description that gives the error but not the move keeps the selection of the model. In the OLMo 2 1B series, the effect is in the first model. The DPO step removes most of it. The source gives no result for large models.

Insight: The software around the model can cause the error

A different test of eight models found an average error rate of 0.649 when software used a tool result again across tasks. The two results come from the data that the control software gives back to the model. This reads as a sign that many agent errors start in the software around the model, and who controls the transcript controls the errors. Teams that make agents must test that software as they test the model.