An attacker can wait for certification and then start an attack

Researchers show that a finite safety test cannot find an adaptive attacker. The attacker behaves like a system with no attack until certification. After certification, it starts the attack. The test cannot find it because the system looks safe during the test. The result covers embodied AI and tool-use AI agents. In AgentDojo trials, the attack caused 1.98 more security violations for each four tasks.

The main result

The attacker copies the behavior of a system with no attack until certification. It keeps access to the system after that. The researchers give a formula for the number of violations after certification.

The tests

• In the RoCo simulation, the attack started after certification and caused separation violations in 34 of 36 episodes.
• In AgentDojo trials, the attack caused 1.98 more violations. The 95% interval is 1.67 to 2.29.
• A language-model attacker found the hidden time of certification from the records of the test.

The defense

The researchers propose a monitor that stays on after certification. It can remove permission for good. This keeps the more violations above a set rate small.

Against an attacker that knows the rule of the monitor, violations fell from 12.00 to 3.81 for each 24 tasks. Removal of permission also decreases the number of tasks that the system does.