Shared examples cut finds of new AI tools from 73% to 40%

Researchers studied AI agents that find tools by search in the tool documents. Agents that share a tool catalog can put usage examples together. These are old requests, each with the tag of the tool that did it. On two of three catalogs, this makes it more difficult to find a tool that no agent used. On StableToolBench, with the primary embedding model, the share falls from 73 to 40%.

How it works

An embedding model changes text into number lists, to match a request to a tool. In folding, each usage example is an added entry for its tool in the search index. The new rule ZoE scores the documents and the use of a tool apart. A tool with no use keeps its document score.

The numbers

• Folding helps tools that have examples.
• ZoE finds new tools more frequently than a router that keeps result slots for use matches. This is true on all three catalogs.
• With 5 simulated agents, folding lowered the finds for each agent.

What is not known

With ZoE, the researchers found no average drop, but some agents continued to lose. The router finds more of the other tools on two catalogs. A simple rule that adds a constant to scores shows no different result from ZoE on new tools.

Insight: Shared memory may favor tools that agents use

An earlier test of tool graph memory gave small increases and used more than 2 times the model calls. This test shows that put together examples help used tools and cause a loss for new ones. Shared use seems to favor tools that agents use now, so a new tool is difficult to find until it has examples. Tool makers get an advantage from first use. Test new tools apart.