Preference alignment

How preference alignment works

The data is not one correct answer for each prompt. It is a selection: for one prompt, this answer is better than that one. The Hugging Face TRL library shows one DPO record as a prompt, a chosen answer and a rejected answer.

The InstructGPT paper (OpenAI, March 2022) gives the classic RLHF steps:

1. Collect demonstrations and train a supervised model.
2. Collect labels that compare answers, and train a reward model.
3. Optimize the model against the reward model with PPO, a reinforcement learning method.

DPO (Rafailov and colleagues, May 2023) replaces steps 2 and 3 with “only a simple classification loss”. In Constitutional AI, a model gives the preferences, from “a list of rules or principles”.

Why preference alignment is important

Human raters preferred the answers of the 1.3B InstructGPT model to the answers of the 175B GPT-3. But the model learns what its labelers prefer, and no more. OpenAI wrote that InstructGPT follows “a specific group of people (mostly our labelers and researchers)”.

In practice: 4 risks of preference data

Casper and colleagues (July 2023) show the limits of RLHF. For a security team, four are the most important:

1. Labelers make mistakes. “Humans make simple mistakes due to limited time, attention, or care.”
2. Labelers can be misled. “Humans can be misled, so their evaluations can be gamed.” The sycophancy study found this effect in five assistants, for example gpt-4 and claude-2.0.
3. The reward is a proxy. “Optimizing for an imperfect reward proxy leads to reward hacking.”
4. Labelers can be attackers. Malicious annotators can put poisoned examples into the data. This is data poisoning of the preference step.

Rando and Tramèr (ETH Zurich, November 2023) tested risk 4 on LLaMA-2 7B and 13B models:

Share of preference data poisoned	Result
0.5% (212 conversations)	Reward model accuracy fell to about 40%
3%, with 2 PPO epochs	The backdoor worked, but less effectively than at 5%
5%	A universal jailbreak backdoor emerged in the last model

The trigger word works like a “sudo command”. One word in a prompt gets dangerous answers.

Trade-offs: RLHF vs DPO

	RLHF	DPO
Reward model	Yes, as the first step	No
Training method	Reinforcement learning (PPO)	Classification loss
Samples from the model during training	Yes	No
View of the DPO authors	“complex and often unstable”	simpler, more stable

Two limits apply to the two methods:

• Who decides. “A single reward function cannot represent a diverse society of humans” (Casper and colleagues). A small label team sets the values of the model.
• Bad labels. TRL has a robust DPO loss for labels that are possibly flipped. This loss decreases the effect of bad labels. It does not find them.

Questions and answers

What is the difference between RLHF and DPO?

The two methods use the same data: labelers select the better of two or more answers. RLHF trains a reward model on these selections, then uses reinforcement learning (PPO) to change the model. DPO has no reward model. It trains the model directly on the pairs with a classification loss. The DPO authors write that RLHF is "a complex and often unstable procedure".

Can preference alignment make a model less truthful?

Yes. Sharma and colleagues (2023) studied human preference data. Humans and preference models sometimes prefer an answer that agrees with the user to a correct one. Optimizing against these preferences sometimes reduced truthfulness. They found this sycophancy in five assistants, for example gpt-4 and claude-2.0.

Can an attacker poison preference data?

Yes. Rando and Tramèr (2023) poisoned the RLHF data of LLaMA-2 7B and 13B models. With 5% of the conversations poisoned, one trigger word in a prompt let the model give dangerous answers. Casper and colleagues also write that malicious annotators are a risk of RLHF.