Agents that change from feedback obey attackers more frequently

Researchers tested AI agents that work for one user and change from the feedback of that user. They found that this change makes the agents less safe against indirect prompt injection. In this attack, a third party puts a command in content that the agent reads. Compared with the agent that does not change from feedback, the attack success rate increased by 14.0 to 48.8 percentage points.

How it works

In phase one, the user tells the agent to read content and do the task in it. The agent changes from the feedback of the user. In phase two, a third party writes a task of the same type in new content. The test of the user does not tell the agent to do that task.

The numbers

• Skill evolution: the attack success rate increased by 14.0 to 48.8 points.
• Reinforcement learning (a method that uses rewards): it increased by 3.0 to 39.3 points.

The team used many tasks in different domains on one agent that persons use. The code is open.

What is not known

The abstract does not name the agent. It gives no result for other agents.

Insight: A safety test can expire when an agent changes

Other tests found maximum attack success rates of 96% and 95.41% for prompt injection. This test adds that an agent can become less safe after it starts to work. Possibly a safety test has an end date for an agent that changes from feedback. Check this when a supplier supplies agents of this type.