In tests, 62 of 72 AI agent incidents with damage had no attacker
A warning that tells the model about the danger after each tool output cut damage for all 5 models. A note of the same length did not.
Researchers made ROGUE-AI, a tool that builds again each reported AI agent incident offline. It also makes a matched control that has not the cause which the report names, and it operates agents on both. It found the reported damage in 89% of 108 tries. Agents on 5 models made the reported mistake in 18% to 33% of tests, against 1% or less without the cause.
How it works
A news report seldom tests the cause. ROGUE-AI uses a group of AI agents to build each incident again. Then it operates test agents on the incident and on the matched control.
The numbers
- The tool found the damage in 89% of 108 tries.
- 72 incidents had damage in the real world, and 62 had no attacker.
- Without an attacker, the tests caused more damage than tests of prompt injection, for all 5 models.
In those 62 incidents, the agents did their task. They did not see a usual danger, such as a database that the live site also uses. Prompt injection is a hidden instruction that an attacker puts in data.
What is not known
The authors give no damage rates for the warning test. They tell that vendors can operate the tests again before each new release.
Insight: Most agent damage has no attacker, so attack filters miss it
In 2026, a test of the Log-To-Leak attack hid an instruction in tool text, and a detector looked for it. ROGUE-AI found no attacker in 62 of 72 incidents with damage. This reads as a gap: defense spend follows the attacker, but the usual mistake is a danger that the agent did not see. Leaders must ask for danger tests, not only attack tests.
Sources
Posted