Incorrect risk warnings made Gemini refuse 86.3% of safe tasks
An attacker can stop an AI agent if the attacker puts an incorrect warning in content that the agent reads.
Researchers studied an attack where incorrect risk warnings in content that is not trusted make an AI agent refuse a safe task. They tested three model families on 51 task pairs. For Gemini, the refusal rate increased from 9.8% to 86.3%, and clean tasks continued to work. Prompts that tell the agent to be careful made the attack stronger.
How it works
The attacker does not make the agent do injury. The attacker puts a made-up warning in a web page or a file, and the agent stops. The team calls this over-abstention (a refusal that is not necessary).
The numbers
The test is small: three model families and 51 task pairs. The words of the warning changed the results a lot, and the results are different for each model. A check named Verify-Before-Abstain compares a warning with trusted records or the content. It increased the number of tasks that agents completed.
What is not known
The check needs trusted records. The paper says that forged records and sensitive content show its limits, with a risk to confidentiality.
Insight: A safety rule can become a way to attack
In October 2026, researchers showed that a delayed message made some Grok setups stop a life-support system in a simulation. This paper shows the other side: attackers make agents stop. This reads as a move to deny service: a rule to stop when not sure gives the attacker control. Leaders must test refusals and also steps.
Sources
Posted
