6 AI agent frameworks operated a tool call from an attacker

Researchers from 3 universities made 6 AI agent frameworks operate tool calls from an attacker. They told the model to write the tool call again, and the framework then operated it. The frameworks include LangGraph, AutoGen and CrewAI. The team also found bugs in 118 of 273 servers in its tests. No other group did a check of the paper.

How it works

The researchers told a model to write a text again. The text was a tool call. The model gave a refusal, but it also wrote the text. LangGraph did not operate the tool call.

Then the researchers told the model that the task was only an example, and to give only the text. The model gave a correct tool call, and LangGraph operated it immediately with the data from the attacker. The model gave its refusal only after the tool gave a result.

Frameworks affected

These 6 frameworks operated a tool call from the attacker:

• LangGraph, from LangChain
• Microsoft Agent Framework
• AutoGen, from Microsoft
• Agent Development Kit, from Google
• Strands Agents, from AWS
• CrewAI

Flaws in tools

The team made ToFu, a test tool that sends test inputs to a tool. It checks if the tool keeps the limits that its description gives. The team used it on 273 Model Context Protocol servers.

The team found bugs in 118 servers. The most dangerous bugs include 64 path-traversal flaws.

What is not known

The paper does not show that this occurs in all systems with the frameworks. It does not tell if the bugs continue to occur, or if attackers used them.