Definition · AI security
AI dataset provenance
AI dataset provenance is a maintained record of where a training dataset came from and every modification made to it since. MITRE ATLAS files it as a mitigation, AML.M0025, because that record is what lets a team establish whether the data a model actually trained on is the data they reviewed and approved.
Last reviewed
Key points
- ATLAS asks for two things, the dataset's source and a complete record of any modifications. At web scale the index of most datasets goes unchanged long after publication, even after much of the data goes stale.
- Provenance is a record, not a scan. It does not inspect what a model learned; it establishes whether the data that reached training is the data that was approved.
- A dataset shipped as a list of URLs has no working provenance until a hash binds each index entry to the content a downloader later retrieves.
- Carlini and colleagues priced that gap rather than exercising it. 60 US dollars a year of expired domains would have bought control of 0.01 percent of LAION-400M or COYO-700M, at July 2023 prices.
- A hash mismatch proves difference, not attack. Of 2.9 million Conceptual Captions images still online in 2023, only 1.1 million still matched their 2018 hashes.
AI dataset provenance is record keeping used as a security control. It counts as one because, NIST says, weights may not be auditable the way source is. The data that went in is where the question stays answerable.
MITRE ATLAS files it as AML.M0025: “a detailed history of datasets used for AI applications” covering “the dataset’s source as well as a complete record of any modifications”.
What the record has to bind to
A provenance record becomes a control when it binds to content. Web scale datasets ship as an index of URLs, so a sample’s name and its bytes are held by different parties.
Carlini and colleagues measured the gap. Very few datasets “include any form of cryptographic integrity check of the downloaded content”, and most indexes go unchanged long after publication even after much of the data goes stale. Buying expired domains, 60 US dollars a year would have bought control of 0.01 percent of LAION-400M or COYO-700M, at July 2023 prices.
NIST’s fix is the binding: “the provider publishes cryptographic hashes, and the downloader verifies the training data”, citing that paper.
Why a mismatch is not a finding
A hash proves difference, not attack, and web content changes for ordinary reasons. Of 3.3 million Conceptual Captions images from 2018, 2.9 million were online in 2023 and only 1.1 million still matched. The defence is perfect against split view poisoning and costs real data.
A hash alone cannot say whether data poisoning happened. ATLAS asks for the modification record as well as the source: the hash raises the question, the history answers it.
Where definitions disagree
ATLAS states the same requirement twice. AML.M0023, AI bill of materials, repeats the provenance sentence almost verbatim inside a broader artifact listing. The filing separates them anyway. Provenance is “Technical - AI” and points at dataset integrity and at datasets masquerading as trusted. The BOM is “Policy” and points mostly at untrustworthy binaries and packages — though ATLAS maps it to training data poisoning too, on the same artifact sentence.
Questions and answers
What is the difference between AI dataset provenance and an AI bill of materials?
Scope and moment. An AI bill of materials is a listing of every artifact that went into a system, read at a point in time. Dataset provenance is the history of one dataset over time, being its source plus a complete record of every modification since. MITRE ATLAS keeps both, states the provenance requirement inside each of them, and then separates them by category and by what they defend. ATLAS files provenance as "Technical - AI" and maps it mostly to data techniques, though it reaches model poisoning too. It files the AI BOM as "Policy" and maps it mostly to artifacts, binaries and packages, though that mapping reaches training data poisoning too.
Why is dataset provenance a security control rather than documentation?
Because the alternative audit may not be available. NIST states that organizations and researchers "may not be able to audit and identify vulnerabilities encoded into a model's weights in the same way it is often possible to audit open-source software". That is a limit rather than an impossibility: NIST also records mechanistic interpretability, used to identify backdoor features, as a current proposed approach to detecting vulnerabilities introduced by model poisoning. Provenance does not wait on that research. It makes the question answerable from the other end, the data that went in, under the conditions in which it is actually asked, which are an incident and a deadline. ATLAS maps it against five techniques, including Training Data Poisoning and Masquerading, where the stated value is identifying datasets falsely presented as trusted.
Do cryptographic hashes make dataset provenance verifiable?
They make the source half verifiable. NIST recommends that "the provider publishes cryptographic hashes, and the downloader verifies the training data", which binds a URL in a dataset index to the bytes that URL served at snapshot time. What a hash cannot do is distinguish an attack from ordinary change. Carlini and colleagues found that of 2.9 million Conceptual Captions images still online in 2023, only 1.1 million still matched their 2018 hashes, so most mismatches in a real dataset are drift. That is the work the modification record does.
Does dataset provenance replace sanitizing the training data?
No, and NIST puts them in an order. Sanitization looks for poisoned samples in the data itself; provenance and integrity attestation protect the dataset once sanitized, so that the reviewed version is the version that reaches training. Provenance on its own attests to a history. It never claims the data in that history was clean to begin with.