A new attack gets AI agent system prompts with 34.3% more function

Researchers made ExtraHerO, an attack that gets the system prompt (the set instructions of an AI system) of chatbots and agents. Known attacks work less on new systems. Their system prompts have more parts, and the attacker sees only the last task step, not the raw text. On agents, ExtraHerO increases fidelity by 12.2%, function by 34.3% and adversarial utility by 30.5% on average.

How it works

The attack makes the system prompt in parts with a set structure template. It changes its query with feedback on the differences from the template. It stops or starts again when new information is small.

The numbers

On chatbots, the average increase is 12.5% in fidelity, 2.3% in function and 6.0% in adversarial utility. Adversarial utility is the help that the system prompt gives to subsequent live attacks. The authors also write that known fidelity metrics can give an incorrect view.

What is not known

The abstract gives no test on a live system from a vendor. It gives no defense.

Insight: A leaked system prompt helps the next attack

The paper writes that the system prompt is very important intellectual property. It holds tool names, logic and safety rules. A related study did tests on the system prompts of 3 agents that write code and found 152 results, at a total cost of $0.27. The two papers find the system prompt as a target, and it reads as a part that attackers will get. Do not put secrets in it.