Small AI agents ignore most attacks and do not refuse them
In 33,000 test runs, 76 to 80% of the attacks that did not work were ignored, so a low attack score is not proof of safety.
Claimed, not confirmed
Researchers checked 33,000 runs of eleven open-weight agents, which are models with public weights. Each agent is a model of 2 to 14 billion parameters, which are internal values. The InjecAgent and AgentDojo tests score an agent by the attack success rate (ASR), which is the share of attacks that work. The researchers show that this score hides how the other attacks end.
How it works
Hidden instructions in external content can get control of the tool calls of an agent. This is indirect prompt injection. Tests count each attack that does not work as a defense that worked. The researchers used a judge model that did not see the labels to sort these attacks.
The numbers
- 76 to 80% of the attacks that did not work were ignored by the agent.
- With the same model weights, a change of the system prompt moved the ASR on InjecAgent from 0.2% to 43%.
- In 21.7% of AgentDojo episodes, the agent was safe but did not complete the task.
What is not known
The study covers only small open-weight agents. The authors do not say if larger or closed models work the same. They release the labeled data, the judge prompt and the code. No other group has checked the result.
Insight: One system prompt can move an attack score
Other tests give single ASR numbers: 32% for Meta SecAlign and 58% for GPT-4o on LLMail-Inject. This study shows that the same weights give 0.2% or 43% with only a new system prompt. A low ASR can come from an agent that ignores the attack. A vendor can then show a good score without a safe system. The buyer has more control when the buyer wants more than the score.
Sources
Posted
