Agents write to incorrect records when a lookup only counts matches
A lookup that showed a customer with the same name first caused incorrect writes in about a quarter of conversations in a retail test.
A paper tests the behavior of an AI agent when a tool shows one record and a 2nd record also fits. When the tool only counted the 2nd record, most of 36 previous agents used the record that they saw. When the tool showed the 2 records, most agents stopped, and the new Claude and GPT models always opened a counted 2nd record.
How it works
The authors set a rule. An agent can change a record only in one of these conditions:
- It sees that each claim behind the change is correct.
- The user answers the claim.
- The user is told while the change is reversible.
The authors change the results of a search: a list, a count or no 2nd record.
The numbers
In a retail test, the authors added a customer with the same name and postal code. A lookup that gave this customer first caused writes to the incorrect account in about a quarter of conversations. A lookup that checks a street that the user knows kept the result near the level for one customer.
What is not known
The paper gives no numbers for tests with a claim from the user. When a user says that a folder is empty, a count of its one item stops agents about as frequently as a list.
Insight: The data that a tool shows an agent sets how safe its writes are
Cohere has a Rerank 5 tool that can give a model only the parts of a document about the query. This paper shows that a record that a lookup does not show can make an agent write to the incorrect account. This reads as a move of agent safety from the model to the tools that supply data, and the team that makes them controls it. Teams that make agents must test each lookup.
Sources
Posted