Post-training

How post-training works: 3 steps

Post-training starts from a pre-trained language model. Each step changes the model from the step before it.

Step	What teaches the model	Example
Instruction tuning	Prompts with ideal responses	Llama 2-Chat, first step
[Preference tuning](../../ai-basics/preference-alignment/)	Preference data or a reward model	Direct Preference Optimization in Tülu 3
Reinforcement learning	A reward signal	RLVR in Tülu 3: a constant reward for a correct result

Why post-training matters: 3 security risks

Parameter count does not give good behavior. In a 2022 OpenAI paper, outputs from a 1.3B parameter InstructGPT model were preferred to outputs from the 175B parameter GPT-3.

The same training also makes 3 risks:

1. Safety can break. In 2023, researchers used 10 examples and less than $0.20 of fine tuning to jailbreak GPT-3.5 Turbo.
2. Backdoors can stay. In a 2024 study, deliberately trained backdoors stayed through standard safety training.
3. Rewards can teach incorrect behavior. A 2023 study links sycophancy in part to human preference judgments.

In practice

“Post-training research” is the work to make these recipes better. A Thinking Machines Lab advert for a research role (read 30 October 2026) calls a recipe “a collection of datasets, training stages, and hyperparameters”.

Source	Date	Steps	Note
InstructGPT	March 2022	Fine-tuning with human feedback	1.3B parameter model
Llama 2-Chat	July 2023	Supervised fine-tuning, then RLHF	A safety-specific reward model
Tülu 3	November 2024	Data curation, supervised fine-tuning, preference tuning, RLVR	Starts from millions of prompts
OpenAI	May 2025	Supervised fine-tuning, then reinforcement learning	Reward signals from different sources

Three facts help when you read about a recipe:

• Same loss. Instruction tuning keeps the pre-training loss. Prompt tokens are masked.
• Smaller learning rate. The rate is usually 10 to 100 times smaller than in pre-training (Lambert, RLHF book).
• Data limit. Lambert writes that around 1M prompts can be enough. More data then helps less.

Trade-offs

Post-training has these limits:

• Shallow safety. A 2024 paper found that safety alignment can apply mainly to the first output tokens. Benign fine-tuning can also jailbreak aligned models.
• Open weights. A 2024 study erased one direction in the activations of 13 open-source chat models. The models then stopped their refusals. See abliterated vs uncensored models.
• Backdoor limits. The authors of the 2024 backdoor study did not estimate how likely such models are.
• Verifiable rewards. RLVR works without a reward model. The result must be one that software can check, such as a math answer.

Where definitions disagree

Sources use the term with 3 meanings:

• Broad. Technically calls post-training “everything that happens after pre-training to make an AI model actually useful.”
• A set of methods. The Tülu 3 paper calls post-training a collection of techniques, with instruction tuning and RLHF in it. The Llama 2 abstract uses “fine-tuned” for this type of work.
• A different subject. Google uses “post-training quantization” for a conversion that can make a model smaller. This conversion is not one of the behavior steps on this page.

Questions and answers

What is the difference between pre-training and post-training?

Pre-training teaches a language model to predict the next token in text. Post-training starts from that model and teaches behavior, with methods such as instruction tuning and RLHF.

Is post-training the same as fine-tuning?

No. Fine-tuning is one method of post-training. Post-training can also use preference tuning and reinforcement learning.