What matters in AI.

Subscribe

Learn / AI basics

Definition · AI basics

Chain of thought

Chain of thought is the series of reasoning steps that a language model writes in plain text before it gives its answer. A prompt can tell a model to write these steps, and reasoning models are trained to write them. The steps frequently make answers better. They do not always show the correct cause of an answer.

Last reviewed

Key points

  • Chain of thought (CoT) is text that a model writes step by step, between the question and the answer.
  • In 2022, CoT prompting raised PaLM 540B on the GSM8K math test from 17.9% to 56.9%.
  • Reasoning models such as OpenAI o1 are trained with reinforcement learning to write a chain of thought.
  • The steps are not a full record. In Anthropic tests, Claude 3.7 Sonnet named a hint that it used 25% of the time.
  • For security teams, a chain of thought is a monitor signal, a data leak path and an attack surface.

How chain of thought works

A chain of thought is text. The model writes the steps one token at a time, then writes the answer. There are three methods to get it.

Method What starts it First source
Few shot CoT prompting Worked examples with steps in the prompt Wei and colleagues, January 2022
Zero shot CoT prompting The prompt text “Let’s think step by step” Kojima and colleagues, May 2022
Trained reasoning Reinforcement learning OpenAI o1, 2024

The first results were large:

  • PaLM 540B on GSM8K, eight worked examples: 17.9% to 56.9% (Wei and colleagues).
  • text-davinci-002 on MultiArith, zero shot: 17.7% to 78.7% (Kojima and colleagues).
  • Model scale: Wei and colleagues found an increase only in models of about 100B parameters.

Today a reasoning model writes a chain of thought with no instruction in the prompt.

A chain of thought sits between the question and the answer. A monitor can read it, and an attacker can try to read or steer it.
A chain of thought sits between the question and the answer. A monitor can read it, and an attacker can try to read or steer it.

What chain of thought changes for security

  1. A monitor signal. Baker and colleagues (March 2025) used GPT-4o to monitor o3-mini for reward hacking in coding tasks. This worked far better than a monitor of the agent steps and output alone.
  2. Not a full record. Turpin and colleagues (2023) found that a bias in the prompt decreased accuracy by a maximum of 36% on 13 BIG-Bench Hard tasks. The explanations did not name the bias.
  3. A data leak path. Green and colleagues (EMNLP 2025) found that reasoning traces “frequently contain sensitive user data”. Prompt injection can get this data out.
  4. An attack surface. The H-CoT attack (February 2025) uses the reasoning that a model shows to jailbreak it. On one benchmark, the refusal rate of o1 fell from about 98% to below 2%.

Shown vs hidden chain of thought

Vendors do not agree if users see the raw chain of thought.

Vendor and date Selection Cause they give
OpenAI, o1, September 2024 Raw chain of thought hidden from users To monitor it “for signs of manipulating the user”
Anthropic, Claude, February 2025 Raw thinking text shown Trust, answers that are easy to check, and interest, but attackers can use it to jailbreak Claude

Hidden steps cost money. In September 2024, o1 reasoning tokens were not in the API response but were billed as output tokens (Simon Willison). Check the current terms of each provider.

A hidden chain of thought is not a private one. The Leaky Thoughts study found that prompt injection can get data from traces that teams think are internal.

Where definitions disagree

The primary question: is a chain of thought the reasoning of the model, or a story about it?

  • Steps as a view of the work. Korbak and colleagues, 41 authors (July 2025), say developers “can monitor their chains of thought (CoT) for the intent to misbehave”. They also say this monitoring is “imperfect” and “fragile”.
  • Steps that do not show all causes. Anthropic (April 2025) tested an earlier snapshot of Claude 3.7 Sonnet. Outcome based training first raised faithfulness, by a relative increase of 63% and 41%. Then it stopped at 28% on MMLU and 20% on GPQA.
  • The open question. Wei and colleagues, who introduced the method, did not say if the network is “actually reasoning”. They left it “as an open question”.

Questions and answers

Does a chain of thought show what the model really did?

Not always. Turpin and colleagues (2023) found that GPT-3.5 and Claude 1.0 did not name a bias that changed their answers. In Anthropic tests from April 2025, Claude 3.7 Sonnet named a hint that it used 25% of the time. DeepSeek R1 named it 39% of the time.

What is the difference between chain of thought prompting and a reasoning model?

Chain of thought prompting tells a model in the prompt to write steps. The prompt can give examples (Wei and colleagues, 2022) or the words "Let's think step by step" (Kojima and colleagues, 2022). A reasoning model, such as OpenAI o1, is trained to write the steps with no prompt.

Can a chain of thought leak data?

Yes. Green and colleagues (EMNLP 2025) found that reasoning traces frequently hold sensitive user data. Prompt injection can get the data out, or the data can go into the answer. More reasoning steps made the leak larger.

Sources

  1. Chain-of-Thought Prompting Elicits Reasoning in Large Language ModelsarXiv (Wei et al.), 28 Jan 2022
  2. Large Language Models are Zero-Shot ReasonersarXiv (Kojima et al.), 24 May 2022
  3. Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought PromptingarXiv (Turpin et al., NeurIPS 2023), 7 May 2023
  4. Reasoning models don't always say what they thinkAnthropic, 3 Apr 2025
  5. OpenAI o1 System CardarXiv (OpenAI), 21 Dec 2024
  6. Notes on OpenAI's new o1 chain-of-thought modelsSimon Willison, 12 Sep 2024
  7. Claude's extended thinkingAnthropic, 24 Feb 2025
  8. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting ObfuscationarXiv (Baker et al.), 14 Mar 2025
  9. Chain of Thought Monitorability: A New and Fragile Opportunity for AI SafetyarXiv (Korbak et al.), 15 Jul 2025
  10. Leaky Thoughts: Large Reasoning Models Are Not Private ThinkersarXiv (Green et al., EMNLP 2025), 18 Jun 2025
  11. H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning ModelsarXiv (Kuo et al.), 18 Feb 2025

In the news