Definition · AI basics
Test data
Test data is a set of examples that a team keeps apart from model training. The team uses it at the end to measure how an AI model works on inputs it did not see. The model does not learn from the test data. A test score is correct only if the test data stays out of training and tuning.
Last reviewed
Key points
- Test data is kept apart from training data and validation data. The team uses it last, to make sure of the result.
- A model that you score on its own training data can get a perfect score and fail on new inputs. This is overfitting.
- If a team tunes a model again and again on the test data, the test leaks into the model.
- If public benchmark questions go into the training data of a model, the score goes up with no new skill. This is benchmark data contamination.
- The EU AI Act defines testing data. For high-risk AI systems, the data must be relevant, representative and as free of errors as possible.
How it works
A team divides its data into three sets before training.
| Set | Task | When the team uses it |
|---|---|---|
| Training data | Fits the learnable parameters | During training |
| Validation data | Tunes the model, finds overfitting | Many times |
| Test data | Gives the last measure | One time, at the end |
The EU AI Act uses the same three terms. Article 3(32) defines testing data as data for “an independent evaluation of the AI system”.
The Google Machine Learning Crash Course gives this sequence:
- Train the model on the training set.
- Measure the model on the validation set.
- Change the model, and do steps 1 and 2 again.
- Make sure of the result on the test set.
The sets do not need to be equal. A scikit-learn example keeps 40% of the data for testing.
Why it matters
A test score is the primary sign that a model works on new inputs. A model tested on its training data “would have a perfect score but would fail to predict anything useful on yet-unseen data”, says scikit-learn.
3 problems make test data less useful:
- Overlap. The test set has copies of training examples. Google tells teams to remove these duplicates.
- Tuning against the test. Knowledge about the test set can then “leak” into the model, says scikit-learn.
- Wear. Google says test sets “wear out” with repeated use.
In practice
Public benchmarks are test data for all AI developers. A language model learns from much of the public web, and benchmark questions are on the web. Xu and colleagues (2024) call the result benchmark data contamination. Models “inadvertently incorporate evaluation benchmark information from their training data”.
Zhang and colleagues at Scale AI measured this in 2024:
- They wrote GSM1k, 1,205 new grade school math problems in the style of the public GSM8k benchmark.
- Some models scored up to 8% lower on the new problems.
- Phi, Mistral and some Llama models “seem to be overfitting GSM8k”. Frontier models showed “little to no evidence of overfitting”.
- All models still “broadly demonstrate generalization” to new problems.
- The authors do not release GSM1k to the public at this time.
Benchmark makers use other controls too. Each task file of the BIG-bench benchmark from Google has a “canary” string. The string is there “to prevent BIG-bench tasks from leaking into web-scraped training data”.
For high-risk AI systems, the EU AI Act makes test data quality a legal duty. Article 10 says training, validation and testing data sets “shall be relevant, sufficiently representative, and to the best extent possible, free of errors and complete in view of the intended purpose”.
Trade-offs
| Choice | Gain | Cost |
|---|---|---|
| More test data | Google asks for enough test data for “statistically significant” results | Less data for training. scikit-learn says a three-way split “drastically” reduces the training samples. |
| Cross validation | No separate validation set | A test set must “still be held out for final evaluation”, says scikit-learn. |
| Public test set | Any team can examine a claim | The questions can go into training data. |
| Private test set, like GSM1k | Less risk of leaks | Other teams cannot examine the results. |
Questions and answers
What is the difference between test data and validation data?
Validation data helps a team tune a model, and the team uses it many times. Test data gives the last measure, and the team uses it at the end. The EU AI Act and scikit-learn keep the two sets apart. Tuning on the test data leaks knowledge of the test into the model.
Why can a benchmark score of a language model be too high?
Public benchmark questions can go into the web text that trains a language model. The model can then remember the answers. In 2024, Zhang and colleagues wrote 1,205 new math problems like those of GSM8k. Some models scored up to 8% lower on the new problems.
Sources
- Regulation (EU) 2024/1689 (AI Act), Article 3: DefinitionsFuture of Life Institute (text of Regulation (EU) 2024/1689), 12 Jul 2024
- Regulation (EU) 2024/1689 (AI Act), Article 10: Data and Data GovernanceFuture of Life Institute (text of Regulation (EU) 2024/1689), 12 Jul 2024
- Cross-validation: evaluating estimator performancescikit-learn
- Datasets, generalization, and overfitting: Dividing the original datasetGoogle for Developers (Machine Learning Crash Course)
- Benchmark Data Contamination of Large Language Models: A SurveyarXiv (Xu et al.), 6 Jun 2024
- A Careful Examination of Large Language Model Performance on Grade School ArithmeticarXiv (Zhang et al., Scale AI), 22 Nov 2024
- BIG-bench repository READMEGoogle (GitHub)
Guides that use this term
Related topics
In the news
- Stanford finds AI text in 38% of Santa Clara court complaints
- AutoPI finds more methods to break AI agents than other test tools
- Cloudflare releases 2 open models that read data from documents
- Test finds that Claude Fable 5.2 knows rare protein sequences
- Microsoft starts Azure HorizonDB for AI apps and agents
- Decision models took 1 in 4 Vercel router requests in October
- A new test shows image safety checks do not find risk in some cultures
- Open model Beveridge beats five large AI models at set decisions
- Google adds 40 rule packages and dry runs to Data Boundary
- Daisytuner gets €6 million to move AI code off Nvidia chips
- A researcher in India made a free Odia AI model that operates offline
- An attack makes text in an image clear only after the image shrinks
- Protegrity releases an open model that finds 69 types of personal data
- ServiceNow tool moves firms off legacy CRM systems
- Google makes a Decisions API with a result in 150 to 250 ms
- Tallinn summit hears that Europe is slow while AI agents move fast
- In 72% of bad AI agent runs, proof of the error was ignored
- Deepfake detectors that read a pulse signal learn the video source
- Mercor raises $750 million at a $20 billion value to test AI agents
- Snowflake Hybrid Tables are as much as 8x faster by default
- ClickHouse buys Xata to give AI agents instant database copies
- GPT-4o does not follow 70.7% of the hard rules in its Model Spec
- AWS adds a web search API for AI agents at $7 for 1,000 queries
- Developers say Claude Opus 5.5 got worse, but a test shows no change
- An open AI model almost matches a top closed one at hacking
- A new test shows AI agents do 40.64% of risky system tasks
- Google adds agent security controls and new CodeMender scan modes
- VeriSpoof finds voice deepfakes at 2.14% error with one model
- A new defense cuts attacks on AI agent tool inputs from 93% to 9%
- Study names 9 types of prompt injection attack
- Stanford finds 44 percent of 50 AI model claims are not correct
- Top AI agent scores 48.3% on a Google test of physical engineering
- Llama 3.3 added 135 security flaws, and developers added 12
- Legora tests its AI agents for law without client data
- Gartner names Microsoft, IBM and ServiceNow agent management leaders
- Marble raises $3.75M to teach AI agents on how staff work
- A new test finds more anti-Muslim bias when AI models reason or act
- A Google team makes a video agent that selects when to speak
- Endor Labs puts security checks in Google Antigravity agents
- Knowledge Catalog of Google reads the metric rules of six tools
- DataHub says its tool lifts AI agent precision to near 100%
- A method with a prior is above other methods when sources do not agree
- Gemini Robotics ER 2 is in public preview on Google Cloud
- IBM gives watsonx Orchestrate a gateway for agents from other firms
- A hint moves AI novelty judges by more than 50 points
- Cadence and Nvidia release Maestro AI models for chip design
- A 2015 network beat two top AI models on skin lesions, 62% to 42%
- Anthropic tells companies to use one agent, not many helper agents
- GPT-5.6-Sol gives confidential facts in 7.08% of answers in a test
- Letta test shows large differences in how 5 AI agent tools learn
- Chat refusals did not stop dangerous tool calls in 10% to 52% of tests
- IBM voice model is number 2 on an open English test
- BenchDoor puts a backdoor in models through benchmark scores
- CarryoverBench shows that a planted attack fires again in 5 rounds
- Mercor says the average pay of its AI experts is $110 an hour
- llm-d 1.0 is a stable open-source system to operate AI models
- A monitor flags 7% of attacks that AI agents divide between them
- Guardrail safety scores change with the policy used for labels
- RedDarwin attack tool gets 100% success on GPT-5.6-Sol
- Small changes in test data change the steps that agent guards stop
- Prompt-injection gates sent more than 91.8% of inputs to a reviewer
- A new test set has 38,675 page pairs for prompt injection