AWS and UC Berkeley release Infinity for 75,000 agents in 8 GB

AWS and UC Berkeley released Infinity as open-source software with an Apache 2.0 license. It is part of their Hydro project. Infinity saves an agent when the agent uses a tool, and then stops it until the tool gives its result. The authors say that about 75,000 agents fit in the 8 GB of a Raspberry Pi 5.

How it works

Most agent tools keep an agent in memory while it waits for a tool. A tool can work for 20 minutes or many days. Infinity saves the conversation and ends the task. It starts the agent again when the tool gives its result. The same method works on serverless platforms, where the function can end during the wait.

The numbers

The test used 80,000 agents for 20 turns each. That is 3.2 million model completions. An agent that waits used about 100 KB of memory. At that rate, 75,000 agents fit in 8 GB. The test used one thread of an AMD EPYC 9R14 processor.

What is not known

The authors say that Infinity is a test release for first feedback. The test used a simulated model and storage in memory. It did not test a model that does work with tools. The authors give no result on cost or speed.

Insight: Free agent tools are a fight for the bottom layer

Infinity is open-source. Perplexity made its SPACE agent sandbox open-source, and AgentR released its free Webcmd memory tool. This reads as a move to control the bottom layer of agents, because the first tool that teams use can set the terms. Free tools can keep a team to one tool. Before you use one, check how hard it is to stop.