Instruction tuning

Two stages, very different sizes

A language model is first pre-trained to predict text across a very large corpus. Instruction tuning is a second pass. Each training example states a task as an instruction and gives the answer wanted. In the FLAN paper, Wei and colleagues tuned a 137-billion-parameter model on “over 60 NLP datasets” phrased this way. Given no worked examples, it beat GPT-3 on 20 of 25 datasets from task types held out of tuning, though the gains appeared only at large model sizes.

The second stage is much smaller. The LIMA authors tuned a 65-billion-parameter LLaMA model on 1,000 examples, “roughly 750,000 tokens”. That model was pre-trained on about 1.4 trillion tokens. LIMA is deliberately small: earlier work used “large multi-million-example datasets”.

Why it matters for poisoning

NIST names instruction tuning as a stage data poisoning can target, “beyond the vast quantities of pre-training data”.

In 2023 Wan and colleagues trained on ten tasks of about 500 samples each and planted 100 poison examples across five of them, roughly one in fifty. A chosen trigger phrase then worked as a backdoor on held-out tasks that were never poisoned, while “poisoning does not affect accuracy on regular inputs”. In that set, a hundred was a small number but not a small share. Web-scale data poisoning is measured the other way, as a percentage of a scraped corpus.

Outsiders can contribute to these sets. Wan and colleagues note that FLAN aggregates open-source datasets, OpenAI drew on prompts users submitted, and “organizations seek to maximize fine-tuning data quantity”. They also note a limit: InstructGPT trains on “a small fraction” of user queries, so an attacker’s submission is unlikely to be picked. Filtering defences measured on such a set are in sanitize training data.

Where definitions disagree

The term overlaps with supervised fine-tuning. MITRE ATLAS lists “Supervised Fine-Tuning (SFT)” and “Instruction Tuning” as separate alignment methods. InstructGPT calls its step of training on labeller-written answers supervised fine-tuning, while LIMA’s supervised fine-tuning on 1,000 prompts is what it calls instruction tuning data. ATLAS also lists instruction tuning under model alignment, a security mitigation.

Questions and answers

What is the difference between instruction tuning and pre-training?

Pre-training teaches a language model by having it predict text across a very large corpus. Instruction tuning comes after it and teaches the model to respond to instructions, using a far smaller set of instruction-and-answer examples. The LIMA authors fine-tuned a 65-billion parameter LLaMA model on 1,000 examples, about 750,000 tokens; that model was pre-trained on about 1.4 trillion tokens. Their results, they write, "strongly suggest" that almost all of a model's knowledge is learned in pre-training.

Why can 100 poisoned examples affect an instruction-tuned model?

Wan and colleagues found that fine-tuned models update quickly: training on 100 examples "can overwrite a model's immense pre-training prior" for a phrase. Their training set was ten tasks of about 500 samples each, so the 100 were roughly one in fifty. With 100 deliberately mislabelled examples, a 3-billion-parameter model labelled 92.8% of negative test inputs containing the trigger phrase "James Bond" as positive, on tasks it was never poisoned on. Larger models were more vulnerable in some settings, not less.

Is instruction tuning the same as RLHF?

No. Reinforcement learning from human feedback is a separate stage that can be run after instruction tuning. Wan and colleagues describe RLHF as conducted "after conducting instruction-tuning", and LIMA was instruction-tuned "without any reinforcement learning".