What matters in AI.

Subscribe

Learn / AI basics

Definition · AI basics

Training data

Training data is the set of examples that an AI model learns from. Training changes the learnable parameters, the weights, until the model fits these examples. The EU AI Act defines training data as data used for "fitting its learnable parameters". Training data sets what the model knows, what it can leak, and what an attacker can plant.

Last reviewed

Key points

  • Training data is the set of examples that changes the weights of a model. Validation data and test data are kept apart from it.
  • A large language model learns first from very large text sets, mostly from the public web. Llama 3 used more than 15 trillion tokens (Meta, April 2024).
  • A smaller, curated set then teaches the model to follow instructions. InstructGPT used about 13,000 prompts for this step (OpenAI, 2022).
  • Training data is an attack surface. An attacker can plant poisoned examples, and can sometimes get training examples back from the model.
  • The EU AI Act sets quality rules for the training data of high-risk AI systems. It also makes general-purpose AI providers publish a summary of it.

How it works

Training changes the model weights until the model fits a set of examples. Those examples are the training data.

A team keeps two other sets apart from it:

Set Changes the weights Use
Training data Yes The model learns from it
Validation data No Tune the model, find overfitting
Test data No The last measure, on new inputs

The EU AI Act uses these three terms in Article 3.

The two steps of training data for a language model

A language model usually learns from training data in two steps:

  1. Pretraining data. Very large text sets, mostly from the public web. Teams use filters and remove copies before use.
  2. Post-training data. Smaller, curated sets. Persons write example answers, or compare answers of the model.
A language model learns first from a large web text set, then from a small curated set.
A language model learns first from a large web text set, then from a small curated set.

Why it matters

Training data sets what a model knows and does. It is also an attack surface. Training data has 3 risks for security teams:

  • Data poisoning. An attacker plants examples that teach the model an incorrect behaviour or a hidden backdoor attack.
  • Extraction. An attacker queries the model and gets training examples back, word for word. This is a risk from memorization.
  • Unknown source. If a team does not know where the data came from, it cannot check the data for poison, personal data or licence problems. See AI dataset provenance.

In practice

What the training data of 3 published models contains

Model Training data Source
GPT-3 (2020) 410 billion tokens of filtered Common Crawl, 60% of the mix. Also WebText2, two book sets and Wikipedia. Brown et al., OpenAI
Llama 3 (2024) More than 15 trillion tokens “from publicly available sources”, seven times the data of Llama 2 Meta
InstructGPT (2022) About 13,000 prompts with example answers, and 33,000 prompts with ranked answers Ouyang et al., OpenAI

Filters remove much of the raw data. For GPT-3, OpenAI filtered 45TB of compressed Common Crawl text down to 570GB. Meta used “heuristic filters, NSFW filters, semantic deduplication approaches, and text classifiers” for Llama 3.

The attack numbers

  • 60 USD. Carlini and colleagues (2023) could have poisoned 0.01% of the LAION-400M or COYO-700M datasets for this cost. Their attacks change web content that a dataset points to, or edit crowd-sourced content immediately before a snapshot. See web scale data poisoning.
  • 250 documents. A study by Anthropic, the UK AI Security Institute and the Alan Turing Institute (October 2025) used this number. It put a backdoor in models from 600M to 13B parameters. The number did not become larger with the model. The backdoor only made the model write gibberish after a special phrase.
  • Hundreds of sequences. Carlini and colleagues (2020) extracted hundreds of verbatim text sequences from GPT-2. Some were in only one document of the training data. Larger models were “more vulnerable than smaller models”.

What the law asks

The EU AI Act sets two types of duty on training data:

  • High-risk AI systems (Article 10). Training, validation and testing data must be “relevant, sufficiently representative, and to the best extent possible, free of errors and complete”. Data governance practices must cover “data collection processes and the origin of data” and an examination for possible biases.
  • General-purpose AI models (Article 53). The provider must publish “a sufficiently detailed summary about the content used for training”, and keep a copyright policy. The European Commission published the template for this summary on 24 July 2025.

Teams that want to record what is in a dataset can use datasheets for datasets. Teams that want to remove bad examples before training can sanitize training data.

Questions and answers

What is the difference between training data and test data?

Training data changes the weights of the model. Test data does not. A team keeps test data apart and uses it at the end. It measures the model on new inputs. The EU AI Act defines the two sets apart, with validation data between them.

Can someone get training data back out of a model?

Sometimes, yes. In 2020, Carlini and colleagues queried GPT-2 and extracted hundreds of text sequences word for word from its training data. Some had names, phone numbers and email addresses. Some sequences were in only one document of the training data.

How much poisoned data is necessary for an attack?

Sometimes a small quantity. Anthropic, the UK AI Security Institute and the Alan Turing Institute did a study in 2025. 250 documents put a backdoor in models from 600M to 13B parameters. The backdoor only made the model write gibberish. The authors do not know if the result holds for larger models.

Sources

  1. Regulation (EU) 2024/1689 (AI Act), Article 3: DefinitionsFuture of Life Institute (text of Regulation (EU) 2024/1689), 12 Jul 2024
  2. Regulation (EU) 2024/1689 (AI Act), Article 10: Data and Data GovernanceFuture of Life Institute (text of Regulation (EU) 2024/1689), 12 Jul 2024
  3. Regulation (EU) 2024/1689 (AI Act), Article 53: Obligations for Providers of General-Purpose AI ModelsFuture of Life Institute (text of Regulation (EU) 2024/1689), 12 Jul 2024
  4. Commission presents template for General-Purpose AI model providers to summarise the data used to train their modelEuropean Commission, 24 Jul 2025
  5. Language Models are Few-Shot LearnersarXiv (Brown et al., OpenAI), 28 May 2020
  6. Introducing Meta Llama 3Meta, 18 Apr 2024
  7. Training language models to follow instructions with human feedbackarXiv (Ouyang et al., OpenAI), 4 Mar 2022
  8. Extracting Training Data from Large Language ModelsarXiv (Carlini et al., USENIX Security 2021), 14 Dec 2020
  9. Poisoning Web-Scale Training Datasets is PracticalarXiv (Carlini et al.), 20 Feb 2023
  10. A small number of samples can poison LLMs of any sizeAnthropic, with the UK AI Security Institute and the Alan Turing Institute, 9 Oct 2025

Guides that use this term

In the news