A monitor external to the model cut AI agent attack success to 1.3%

Researchers compared methods to stop prompt injection in AI agents that use tools. Prompt injection is an attack in which text in the data that an agent reads gives it commands. They tested defences in the prompt against rules that software enforces external to the model. With 2 open-weight models, the models proposed the tool call from the attacker in 64% and 84% of attack episodes. The external monitor admitted 1.3% and 7.7%.

How it works

The team treats the model as a part that it cannot trust. A monitor external to the model checks each tool call against the task. They also tested human approval and taint tracking, which marks data from untrusted sources. A human layer with no monitor does not work when an attacker sends many approval messages.

The numbers

• In simulation with a continuous attacker, each prompt defence reaches 100% attack success.
• The task-bound monitor holds dangerous tasks at 0%.
• A naive monitor admits 18 of 33 attack variants. A capability-bound monitor admits 1.

What is not known

Tests with actual models include only 2 small open-weight models and static attacks. The human approval and taint results are from simulation only. The 100% result is from an assumed erosion law, and the detector stack of the larger model contradicts that law.

Insight: Attackers use the model, so checks must be external to it

Papers from before show that attackers get through checks when they use the output of a model or its reasoning style. This paper finds that prompt defences do not work against a continuous attacker, but fixed rules external to the model hold. A check that tells the model to judge is a check that the attacker can steer. It reads as if vendors of rule enforcement are in a strong position.