What matters in AI.

Subscribe

Learn / AI basics

Definition · AI basics

Test data

Test data is a set of examples that a team keeps apart from model training. The team uses it at the end to measure how an AI model works on inputs it did not see. The model does not learn from the test data. A test score is correct only if the test data stays out of training and tuning.

Last reviewed

Key points

  • Test data is kept apart from training data and validation data. The team uses it last, to make sure of the result.
  • A model that you score on its own training data can get a perfect score and fail on new inputs. This is overfitting.
  • If a team tunes a model again and again on the test data, the test leaks into the model.
  • If public benchmark questions go into the training data of a model, the score goes up with no new skill. This is benchmark data contamination.
  • The EU AI Act defines testing data. For high-risk AI systems, the data must be relevant, representative and as free of errors as possible.

How it works

A team divides its data into three sets before training.

Set Task When the team uses it
Training data Fits the learnable parameters During training
Validation data Tunes the model, finds overfitting Many times
Test data Gives the last measure One time, at the end

The EU AI Act uses the same three terms. Article 3(32) defines testing data as data for “an independent evaluation of the AI system”.

The Google Machine Learning Crash Course gives this sequence:

  1. Train the model on the training set.
  2. Measure the model on the validation set.
  3. Change the model, and do steps 1 and 2 again.
  4. Make sure of the result on the test set.

The sets do not need to be equal. A scikit-learn example keeps 40% of the data for testing.

Why it matters

A test score is the primary sign that a model works on new inputs. A model tested on its training data “would have a perfect score but would fail to predict anything useful on yet-unseen data”, says scikit-learn.

3 problems make test data less useful:

  • Overlap. The test set has copies of training examples. Google tells teams to remove these duplicates.
  • Tuning against the test. Knowledge about the test set can then “leak” into the model, says scikit-learn.
  • Wear. Google says test sets “wear out” with repeated use.

In practice

Public benchmarks are test data for all AI developers. A language model learns from much of the public web, and benchmark questions are on the web. Xu and colleagues (2024) call the result benchmark data contamination. Models “inadvertently incorporate evaluation benchmark information from their training data”.

Zhang and colleagues at Scale AI measured this in 2024:

  • They wrote GSM1k, 1,205 new grade school math problems in the style of the public GSM8k benchmark.
  • Some models scored up to 8% lower on the new problems.
  • Phi, Mistral and some Llama models “seem to be overfitting GSM8k”. Frontier models showed “little to no evidence of overfitting”.
  • All models still “broadly demonstrate generalization” to new problems.
  • The authors do not release GSM1k to the public at this time.

Benchmark makers use other controls too. Each task file of the BIG-bench benchmark from Google has a “canary” string. The string is there “to prevent BIG-bench tasks from leaking into web-scraped training data”.

For high-risk AI systems, the EU AI Act makes test data quality a legal duty. Article 10 says training, validation and testing data sets “shall be relevant, sufficiently representative, and to the best extent possible, free of errors and complete in view of the intended purpose”.

Trade-offs

Choice Gain Cost
More test data Google asks for enough test data for “statistically significant” results Less data for training. scikit-learn says a three-way split “drastically” reduces the training samples.
Cross validation No separate validation set A test set must “still be held out for final evaluation”, says scikit-learn.
Public test set Any team can examine a claim The questions can go into training data.
Private test set, like GSM1k Less risk of leaks Other teams cannot examine the results.

Questions and answers

What is the difference between test data and validation data?

Validation data helps a team tune a model, and the team uses it many times. Test data gives the last measure, and the team uses it at the end. The EU AI Act and scikit-learn keep the two sets apart. Tuning on the test data leaks knowledge of the test into the model.

Why can a benchmark score of a language model be too high?

Public benchmark questions can go into the web text that trains a language model. The model can then remember the answers. In 2024, Zhang and colleagues wrote 1,205 new math problems like those of GSM8k. Some models scored up to 8% lower on the new problems.

Sources

  1. Regulation (EU) 2024/1689 (AI Act), Article 3: DefinitionsFuture of Life Institute (text of Regulation (EU) 2024/1689), 12 Jul 2024
  2. Regulation (EU) 2024/1689 (AI Act), Article 10: Data and Data GovernanceFuture of Life Institute (text of Regulation (EU) 2024/1689), 12 Jul 2024
  3. Cross-validation: evaluating estimator performancescikit-learn
  4. Datasets, generalization, and overfitting: Dividing the original datasetGoogle for Developers (Machine Learning Crash Course)
  5. Benchmark Data Contamination of Large Language Models: A SurveyarXiv (Xu et al.), 6 Jun 2024
  6. A Careful Examination of Large Language Model Performance on Grade School ArithmeticarXiv (Zhang et al., Scale AI), 22 Nov 2024
  7. BIG-bench repository READMEGoogle (GitHub)

Guides that use this term

In the news