What matters in AI.

Subscribe

The text check does not find 8 of 19 attacks between AI agents

The same tool also gives an incorrect alarm on 70 of 192 safe replies.

Researchers wrote a paper on DonkeyRail, a public tool that finds agent text that is near a secret attack command. They used it in a test where three AI agents send email to each other. With its default configuration, the tool did not find 8 of 19 outputs that have the attack effect. It also gave an alarm on 70 of 192 safe replies.

How it works

The tool gives a score for how near the words of an output are to the attack text. The authors say an output can keep the effect of an attack when its words change. Safe quotes can also have the same words as the source.

The numbers

  • A tuned tool found 1 output with the attack effect on new templates.
  • It gave 15 incorrect alarms on the same templates.
  • The authors find a threshold error in all 21 public tool configurations.
  • Each complete quotation reply caused an incorrect alarm.

What is not known

The authors used one test setup. They say that different policies can give different safety decisions on the same text. A tool that reads only text cannot select the correct one.

Insight: Attackers change the text, but the checks look at the text

A paper shows that a text check does not find an attack when its words change. A paper of 9 November 2026 found that text in the style of a model got a success rate of 50% at most. These two attacks change the text only, but the checks look at the text, and the next move seems to be checks of the work of agents. Leaders can tell vendors to show these tests.

Sources

  1. Safe Resemblance, Silent Propagation: Limits of Text-Similarity Guardrails for AI Agent Safetyarxiv.org
AI MATTER · NEWS · AI MATTER · NEWS ·17 NOV2026

Posted

Tags

Learn the terms in this story

More in Benchmarks

All Benchmarks news