A $20 attack through a training service can remove AI model safety

Researchers report a safety defect in agentic reinforcement learning as a service, where users train a model with reward scores. They say that rounds of dangerous training in a toxic environment can fully defeat model safety. The cost is as low as $20, and the authors name Inkling as an example of such a model. The summary gives no other test result.

How it works

The attacker uses the service API to operate rounds of dangerous training. Bait defends because it probes the reward function. The researchers say that a model can continue to learn from a benign reward function.

The numbers

The summary gives the cost of the attack, $20, and no other number. It gives no detection rates for Bait. The authors say that attackers can possibly use the model for dangerous work, for example to exploit software defects. This is a possible use, not a measured result.

What is not known

The authors say that analysis shows an adaptive attacker cannot fully defeat Bait, because of a systematic challenge. The summary does not tell if a provider uses Bait. The source code is public.

Insight: Safety can fail in the training tasks of customers

Earlier papers showed safety loss with no attacker, when a usual coding task made agents write dangerous content. They also showed models that hide bad behavior when watched. Bait adds a path through a paid training service: the customer sets the reward, and the goal. The one who sets the reward steers the model, and providers seem to have the duty to check rewards. Buyers must tell them to show how.