Adaptive attack

How it works

Carlini and colleagues define adaptive adversaries as ones “adapted to the specific details of the defense” that try to invalidate its robustness claims. The term comes from general security, for an attacker who “adapts to what the defender has done”.

Tramèr and colleagues describe how they attacked adversarial example defences. In most cases:

1. Read the paper and the source code, looking for reasons the defence might still be beaten.
2. Write a score for the attack to push on (a loss function) that measures progress against this defence and is easy to optimise.
3. Pick a way to optimise that score, run the attack, learn from what fails, and repeat.

Their advice is to keep it simple: attack the full defence end to end, and where a defence has many parts, look for the one or two that matter. Make sure a better score always means a stronger attack.

A common reason a defence needs this is gradient masking. The defence breaks the gradient that standard attacks follow. An adaptive attacker who knows this can recover the gradient another way.

Why it matters

Carlini and colleagues write that an evaluation against non-adaptive attacks “is of very limited utility”, and that with existing attacks at default settings “there is a good chance these attacks will work sub-optimally or even fail against a new defense”.

Tramèr and colleagues studied 13 defences from ICLR, ICML and NeurIPS. Nearly all had run their own adaptive evaluation, often by reusing an attack that had broken an earlier defence. All 13 were circumvented. The tools for a stronger attack already existed. The problem, they conclude, was “methodological rather than technical”: evaluators did not tailor them to the defence.

The same pattern has reached language models. Nasr and colleagues tested 12 recent defences against jailbreaking and prompt injection. Most had reported near-zero attack success rate, tested against static sets of attack prompts or weak optimisation attacks. Adaptive attacks pushed success above 90% for most of them.

In practice

NIST’s guidance on adversarial machine learning says mitigations against adversarial examples “need to be evaluated against strong adaptive attacks”. Its definition of a white-box attack covers adaptive attacks, “in which knowledge of the mitigations applied to the model or the system is explicitly tracked”.

Whether adaptive attacks can be automated is open. Tramèr and colleagues found no single strategy worked on every defence, and state that adaptive attacks “cannot be automated”. Against LLM defences, Nasr and colleagues found automated search and reinforcement learning attacks can systematically bypass defences, though they call them less reliable than attacks like PGD on images. On a selected subset, their search attack succeeded 69% of the time and human red-teamers collectively succeeded every time. They encourage industry labs to keep up human AI red teaming, with the red-teamers given details of any defences in the system.

Tramèr and colleagues also warn against copying their attacks as templates. For any fixed attack, “it is possible to build a non-robust defense that prevents that attack”.

Where definitions disagree

The phrase is used in two different senses.

Adapted to the defence. Carlini, Tramèr, Nasr and NIST mean an attack designed for one specific defence, by an attacker who knows how it works.

Adapted to the input. Wei, Haghtalab and Steinhardt’s jailbreak study used “adaptive attack” for an adversary who “can choose the attack based on the prompt”. It counted a success “if any of the 28 evaluated attacks succeed”. That is picking the best of a fixed menu for each prompt, not designing a new attack against a defence. It still made the point: on the authors’ curated set of 32 harmful prompts, Claude v1.3 resisted every roleplay attack, yet was “100% vulnerable to an adaptive attack”.

When a paper reports results against an adaptive attack, check which one it means.

Questions and answers

What is the difference between an adaptive attack and a white-box attack?

A white-box attack describes what the attacker knows: the model, its weights, its training. An adaptive attack describes what the attacker does with knowledge of the defence: change the attack to beat it. NIST says its white-box definition "encompasses the notion of adaptive attacks". An adaptive attack can still use only queries, as when Nasr and colleagues' human red-teamers beat defences with limited information and feedback.

Can adaptive attacks be automated?

Not fully. Tramèr and colleagues found no single strategy worked on all 13 defences they circumvented and conclude that adaptive attacks "cannot be automated". Nasr and colleagues found automated search and reinforcement learning attacks promising against LLM defences, but on a selected subset the human red-teamers, taken together, still did better.