Definition · AI basics
Training data
Training data is the set of examples that an AI model learns from. Training changes the learnable parameters, the weights, until the model fits these examples. The EU AI Act defines training data as data used for "fitting its learnable parameters". Training data sets what the model knows, what it can leak, and what an attacker can plant.
Last reviewed
Key points
- Training data is the set of examples that changes the weights of a model. Validation data and test data are kept apart from it.
- A large language model learns first from very large text sets, mostly from the public web. Llama 3 used more than 15 trillion tokens (Meta, April 2024).
- A smaller, curated set then teaches the model to follow instructions. InstructGPT used about 13,000 prompts for this step (OpenAI, 2022).
- Training data is an attack surface. An attacker can plant poisoned examples, and can sometimes get training examples back from the model.
- The EU AI Act sets quality rules for the training data of high-risk AI systems. It also makes general-purpose AI providers publish a summary of it.
How it works
Training changes the model weights until the model fits a set of examples. Those examples are the training data.
A team keeps two other sets apart from it:
| Set | Changes the weights | Use |
|---|---|---|
| Training data | Yes | The model learns from it |
| Validation data | No | Tune the model, find overfitting |
| Test data | No | The last measure, on new inputs |
The EU AI Act uses these three terms in Article 3.
The two steps of training data for a language model
A language model usually learns from training data in two steps:
- Pretraining data. Very large text sets, mostly from the public web. Teams use filters and remove copies before use.
- Post-training data. Smaller, curated sets. Persons write example answers, or compare answers of the model.
Why it matters
Training data sets what a model knows and does. It is also an attack surface. Training data has 3 risks for security teams:
- Data poisoning. An attacker plants examples that teach the model an incorrect behaviour or a hidden backdoor attack.
- Extraction. An attacker queries the model and gets training examples back, word for word. This is a risk from memorization.
- Unknown source. If a team does not know where the data came from, it cannot check the data for poison, personal data or licence problems. See AI dataset provenance.
In practice
What the training data of 3 published models contains
| Model | Training data | Source |
|---|---|---|
| GPT-3 (2020) | 410 billion tokens of filtered Common Crawl, 60% of the mix. Also WebText2, two book sets and Wikipedia. | Brown et al., OpenAI |
| Llama 3 (2024) | More than 15 trillion tokens “from publicly available sources”, seven times the data of Llama 2 | Meta |
| InstructGPT (2022) | About 13,000 prompts with example answers, and 33,000 prompts with ranked answers | Ouyang et al., OpenAI |
Filters remove much of the raw data. For GPT-3, OpenAI filtered 45TB of compressed Common Crawl text down to 570GB. Meta used “heuristic filters, NSFW filters, semantic deduplication approaches, and text classifiers” for Llama 3.
The attack numbers
- 60 USD. Carlini and colleagues (2023) could have poisoned 0.01% of the LAION-400M or COYO-700M datasets for this cost. Their attacks change web content that a dataset points to, or edit crowd-sourced content immediately before a snapshot. See web scale data poisoning.
- 250 documents. A study by Anthropic, the UK AI Security Institute and the Alan Turing Institute (October 2025) used this number. It put a backdoor in models from 600M to 13B parameters. The number did not become larger with the model. The backdoor only made the model write gibberish after a special phrase.
- Hundreds of sequences. Carlini and colleagues (2020) extracted hundreds of verbatim text sequences from GPT-2. Some were in only one document of the training data. Larger models were “more vulnerable than smaller models”.
What the law asks
The EU AI Act sets two types of duty on training data:
- High-risk AI systems (Article 10). Training, validation and testing data must be “relevant, sufficiently representative, and to the best extent possible, free of errors and complete”. Data governance practices must cover “data collection processes and the origin of data” and an examination for possible biases.
- General-purpose AI models (Article 53). The provider must publish “a sufficiently detailed summary about the content used for training”, and keep a copyright policy. The European Commission published the template for this summary on 24 July 2025.
Teams that want to record what is in a dataset can use datasheets for datasets. Teams that want to remove bad examples before training can sanitize training data.
Questions and answers
What is the difference between training data and test data?
Training data changes the weights of the model. Test data does not. A team keeps test data apart and uses it at the end. It measures the model on new inputs. The EU AI Act defines the two sets apart, with validation data between them.
Can someone get training data back out of a model?
Sometimes, yes. In 2020, Carlini and colleagues queried GPT-2 and extracted hundreds of text sequences word for word from its training data. Some had names, phone numbers and email addresses. Some sequences were in only one document of the training data.
How much poisoned data is necessary for an attack?
Sometimes a small quantity. Anthropic, the UK AI Security Institute and the Alan Turing Institute did a study in 2025. 250 documents put a backdoor in models from 600M to 13B parameters. The backdoor only made the model write gibberish. The authors do not know if the result holds for larger models.
Sources
- Regulation (EU) 2024/1689 (AI Act), Article 3: DefinitionsFuture of Life Institute (text of Regulation (EU) 2024/1689), 12 Jul 2024
- Regulation (EU) 2024/1689 (AI Act), Article 10: Data and Data GovernanceFuture of Life Institute (text of Regulation (EU) 2024/1689), 12 Jul 2024
- Regulation (EU) 2024/1689 (AI Act), Article 53: Obligations for Providers of General-Purpose AI ModelsFuture of Life Institute (text of Regulation (EU) 2024/1689), 12 Jul 2024
- Commission presents template for General-Purpose AI model providers to summarise the data used to train their modelEuropean Commission, 24 Jul 2025
- Language Models are Few-Shot LearnersarXiv (Brown et al., OpenAI), 28 May 2020
- Introducing Meta Llama 3Meta, 18 Apr 2024
- Training language models to follow instructions with human feedbackarXiv (Ouyang et al., OpenAI), 4 Mar 2022
- Extracting Training Data from Large Language ModelsarXiv (Carlini et al., USENIX Security 2021), 14 Dec 2020
- Poisoning Web-Scale Training Datasets is PracticalarXiv (Carlini et al.), 20 Feb 2023
- A small number of samples can poison LLMs of any sizeAnthropic, with the UK AI Security Institute and the Alan Turing Institute, 9 Oct 2025
Guides that use this term
Related topics
In the news
- OpenAI makes model resale platforms get its approval first
- Anthropic releases Claude Fable 5.2 at a 20 percent lower price
- Humyn Labs raises $47 million to sell robot training data
- Test finds that Claude Fable 5.2 knows rare protein sequences
- Vals test finds AI models mark civilians as targets in a game
- LanceDB gets $57.5M to cut AI training data work
- New score finds the training data most open to privacy attacks
- New turn in OpenAI copyright lawsuit as judge allows 56 briefs
- Deepfake detectors that read a pulse signal learn the video source
- Sycophancy in AI models starts in the first training steps
- A small task change cuts one coding model score from 74% to 53%
- OpenAI tells that its Astra model does 40% less work on safety tasks
- A 2 billion parameter model turns documents into JSON fields
- VeriSpoof finds voice deepfakes at 2.14% error with one model
- The method of 4-bit compression changed a hidden trait in an AI model
- A study finds GPT-4o seems to know books behind a paywall
- A study shows where models hold signs of their training data
- Apriel Voice 27B Preview has 65.51% task success in voice tests
- A small text model aligns more with image AI than a 72B model
- A team of 5 small models solved 49 of 50 hard code runs
- Claude wrote training data that taught AI models to deceive
- A deepfake detector with averaged weights reaches 97.04% accuracy
- Two Stanford authors sue Anthropic and want $150,000 a book
- The best open AI image detector scores 75.0% and the worst 37.5%
- Pronto founder launches Provision Labs to collect robot training data
- Moonshot AI removes attack data from Kimi K3.1 Pro
- A 2015 network beat two top AI models on skin lesions, 62% to 42%
- Clark moves the date for AI that makes a successor to 2027
- Maxinsights gets $30M to record work for robot training
- Google bids $10 million for the data of a bankrupt airline
- Hugging Face gives out Carbon-X, a DNA model for a laptop
- NVIDIA expands Cosmos with 5 data blueprints for physical AI
- A prompt injection detector does not work in agent tests
- Authors say Yuvion VL beats closed models on safety, with no scores
- BenchDoor puts a backdoor in models through benchmark scores
- Mercor says the average pay of its AI experts is $110 an hour
- Guardrail safety scores change with the policy used for labels
- A small Qwen3 model cuts AI writing tics from a score of 121 to 12.5
- MetaExp adds 14.8 points on AutomationBench to an agent model
- Zephon data loader gives the same results on 8 to 64 GPUs
- Prompt-injection gates sent more than 91.8% of inputs to a reviewer
- COMPASS finds if an image was in the training data of a model