Letta test shows large differences in how 5 AI agent tools learn
Letta Dream, the tool of the company that made the test, got 73.3%. The scores of the 5 harnesses went as low as 26.7%.
Claimed, not confirmed
Letta released FrontierMemory, a test of if AI agents learn from work done before. 5 agent harnesses, which are the software that operates an agent, did the test. The scores went from 26.7% to 73.3%. The same company, Letta, made the test and the top harness. Letta says that tests of facts do not show if memory helps in new work.
How the test works
Agents get transcripts of work done before, with good and bad tries. They write their memories from them. Then they use the memories on new tasks. Letta says that the harness can be as important as the model in it.
The numbers
Letta also examined 7 AI models for the quality of the memories they write. Claude Opus 5.5 and GLM-5.3 had the same top score.
The price of the OpenAI model is about $0.77 for each task, against $2.80 for Opus 5.5.
What is not known
The test has 15 tasks and one try for each. One result can change a score by about 6.7 points. No other group examined the test. Without checks on the data that agents keep, they can learn bad habits.
Insight: Memory claims come from the sellers
Databricks says that memory gave 3 times more correct results in its tests. The test of Letta puts its tool first. This seems to be a race to set the measure of memory. The maker of the test sets the rules. One more test found that agent memory keeps signs of secret data. Test memory on your tasks.
Sources
Posted
