Model card

A model card is a short document that travels with a trained model and says what it is for, how well it works and where it breaks. Mitchell et al. proposed the format at FAT* in 2019 as “short (one to two page) records”, the complement to a datasheet for a dataset, which covers the data going in rather than the model coming out.

How it works

A model card has nine sections: model details, intended use, factors, metrics, evaluation data, training data, quantitative analyses, ethical considerations, and caveats and recommendations. The weight sits in quantitative analyses, which “should be disaggregated, that is, broken down by the chosen factors” — results per group and per intersection of groups, such as age and skin type, not one headline accuracy number.

Why it matters

A model card is a convention, not a contract, and the paper’s own qualifications say so. Its sections are “not intended to be complete or exhaustive, and may be tailored depending on the model, context, and stakeholders”. Its training data section carries the note “May not be possible to provide in practice” — the gap an AI bill of materials exists to close.

In practice

Google’s TensorFlow Model Card Toolkit is archived, with no push since July 2023, and the format survived by moving into the bill of materials specifications. CycloneDX’s model card schema “was heavily influenced by Tensorflow ModelCard Toolkit and specifically its ModelCard fields”, and CycloneDX now carries a modelCard object that SHOULD be specified for any machine-learning-model component. SPDX’s AI profile covers the same ground.

Absorbed is not structured. SPDX defines limitation and modelExplainability as “free-form text”, and CycloneDX describes the model card a Hugging Face repository publishes as “mostly unstructured information in the form of a markdown file”.

Where definitions disagree

Whether a model card is an AIBOM. CycloneDX’s guide argues the EU AI Act “effectively endorses moving away from the current non-normative publication of model cards and research papers” towards “normative and standardized methods such as AI/ML Bills-of-Materials” — a standards body’s reading, not the Act’s words. The distinction that holds is the consumer, not the content. Mitchell et al. wrote for people, counting “Impacted individuals” among the stakeholders. A bill of materials is queried across a fleet, and a markdown file with the right headings does not answer which deployed systems inherited a given checkpoint.

Questions and answers

What goes in a model card?

Mitchell et al. propose nine sections — model details, intended use, factors, metrics, evaluation data, training data, quantitative analyses, ethical considerations, and caveats and recommendations. The quantitative analyses section is the one that distinguishes the format, because the paper asks for results "disaggregated, that is, broken down by the chosen factors" rather than a single accuracy figure. The paper is explicit that the list is "not intended to be complete or exhaustive, and may be tailored depending on the model, context, and stakeholders", so a model card is a reporting convention rather than a schema to validate against.

Is a model card the same as an AI bill of materials?

No, and the difference is the consumer rather than the content. A model card is written to be read — Mitchell et al. count "Impacted individuals" among its stakeholders — while an AI bill of materials is queried across a fleet of deployed systems. The BOM formats did take the model card's fields. CycloneDX defines a modelCard object that SHOULD be specified for any machine-learning-model component, and SPDX's AI profile covers similar ground. But SPDX defines limitation and modelExplainability as "free-form text", and CycloneDX describes the model card a Hugging Face repository publishes as "mostly unstructured information in the form of a markdown file". A field in a schema is not automatically a field you can query.

Are model cards required by law?

Not as a binding duty. Article 53(1) of the EU AI Act requires providers of general-purpose AI models to keep technical documentation of the model "including its training and testing process and the results of its evaluation", and to publish "a sufficiently detailed summary about the content used for training". Neither provision names a model card. The Act uses the term once, in recital 89, which encourages developers of free and open-source components "other than general-purpose AI models" to adopt "widely adopted documentation practices, such as model cards and data sheets" — a recital rather than an article, and a different population from the one Article 53 binds. CycloneDX's guide reads the Act as endorsing a move away from "the current non-normative publication of model cards and research papers" towards machine-readable bills of materials, but that is a standards body reading a regulation. A model card remains a voluntary convention, which is why its content varies so widely between publishers.