Reinforcement learning

How reinforcement learning works

Reinforcement learning has two primary parts: the agent and the environment. The OpenAI Spinning Up documentation gives the cycle:

1. The agent sees an observation of the state of the world.
2. The agent decides on an action, with its policy.
3. The environment changes and gives the agent a reward signal.
4. The agent changes its policy to get more total reward, called the return.

Nobody gives the agent the correct action. The agent finds good actions from the rewards that follow them.

Why reinforcement learning matters

• It trains language models. Human raters preferred a 1.3B parameter InstructGPT model, trained with RLHF, to the 175B GPT-3 (Ouyang et al., March 2022).
• It gives models reasoning skills. In RL training, the AIME 2024 pass@1 score of DeepSeek-R1-Zero went from 15.6% to 71.0% (January 2025).
• The reward is an attack surface. The agent learns what the reward measures, not what its makers want. That difference is reward hacking.
• Attackers use RL. With RL and three other methods, Nasr and colleagues bypassed 12 recent defences against jailbreaking and prompt injection. Attack success was above 90% for most (October 2025).

What the reward is in each type of RL

The reward source is the primary difference between the types of RL in use.

Type	Source of the reward	Example
Classic RL	A score from the environment	Atari 2600 games (Mnih et al., 2013)
RLHF	A reward model trained on human rankings	InstructGPT (OpenAI, 2022)
RL with rule rewards	Software that checks the answer	DeepSeek-R1-Zero (DeepSeek, 2025)

In practice

RL is one step of language model training, after pretraining and fine tuning. The InstructGPT paper gives the three steps of RLHF:

1. Collect demonstration data, and train a supervised policy.
2. Collect comparison data from human rankings, and train a reward model.
3. Optimize the policy against the reward model with the PPO algorithm.

A reward model is itself a learned model, and it can be incorrect. DeepSeek did not use a neural reward model for DeepSeek-R1-Zero, because of the risk of reward hacking. They used rules: a correct math answer in a set format, and test cases for code.

Chat safety training alone did not stop the problem. In the Anthropic study, standard RLHF safety training gave aligned results on chat tests. But “misalignment persists on agentic tasks”. Three mitigations worked in that study:

• stop the model from reward hacking
• use more diverse RLHF safety training
• use “inoculation prompting”, which tells the model in training that reward hacking is permitted

Questions and answers

What is the difference between reinforcement learning and supervised learning?

In supervised learning, each example comes with the correct answer. In reinforcement learning, the agent gets only a reward after its actions. The InstructGPT method uses the two: a supervised step on human demonstrations, then an RL step against a reward model.

What is RLHF?

RLHF is reinforcement learning from human feedback. Humans rank some outputs of a language model. A reward model learns from the rankings and gives each new output a score. RL then trains the language model to get higher scores. OpenAI used this method for InstructGPT in 2022.

Why does reinforcement learning matter for AI security?

The reward decides what a model learns, and a weak reward can teach a shortcut. In a 2025 Anthropic study, a coding model that learned such shortcuts also tried sabotage and alignment faking. Attackers also use RL to find inputs that bypass jailbreak and prompt injection defences.