Model weights

How model weights work: 3 steps

1. Training sets the weights. The developer trains the model on many examples. The examples set the values of the weights, and thus the behaviour of the model (NTIA, July 2024).
2. The weights go into files. After training, a model is a set of large files of numbers. A model checkpoint of a training run and a released model are two examples.
3. Inference uses the weights. Software loads the files and multiplies each input by the weights to make an output. The Google glossary: inference uses “those learned weights to make predictions”. Inference does not change them.

Weights and biases together are the parameters of a model. The parameters page tells how to read a count such as 70B and how much memory it needs.

Why model weights are a security risk

The weights are the model. A person can use the weights without the training data or the source code. NTIA lists what a person with the weights can do:

• Run the model on their own computers.
• Change the model with fine tuning on new data.
• Remove safeguards. NTIA cites a 2023 study that removed the safety features of Llama 2-Chat 13B for less than $200.

The file is also a risk to the person who loads it. Pickle “is the default format for PyTorch model weights”, Hugging Face writes, and a pickle file can run code when it loads. PyTorch says not to load data from an untrusted source. Hugging Face says its pickle scanner is not 100% foolproof.

In practice

3 risks of model weights and their controls

Risk	Example	Control
Theft or leak of closed weights	LLaMA (2023) and Miqu (2024) leaks, below	Access controls, egress limits, two-person access
Code in a weights file	A pickle file runs code at load	Safetensors, weights_only=True in torch.load, trusted sources
Misuse of released weights	Safeguards removed with fine-tuning	None after release. NTIA: a release “cannot feasibly be reversed”

2 leaks of model weights

• LLaMA, March 2023. Meta gave LLaMA (7B to 65B) to approved researchers on 24 February 2023, with a noncommercial license. About one week after that, a 4chan user posted a BitTorrent link to the full package. Meta sent takedown requests.
• Miqu, January 2024. A model with the name miqu-1-70b appeared on Hugging Face and on 4chan. Mistral CEO Arthur Mensch said that “an over-enthusiastic employee of one of our early access customers” leaked “a quantised (and watermarked) version” of an old model.

How AI labs keep weights safe from theft

• RAND report, May 2024. Irregular, the company of a co-author, writes that the report finds “38 meaningfully distinct attack vectors”. Irregular writes that the report defines five security levels, SL1 to SL5, for attackers from criminals to nation states.
• Anthropic, May 2025. The ASL-3 Security Standard uses “more than 100 different security controls”. Examples are two-party authorization for weight access and egress bandwidth controls. Anthropic writes that it is “suitable against sophisticated non-state attackers”.

Weights vs source code vs training data

The phrase “model weights and source code” names two different things. NTIA notes that developers can release weights, code, data and documentation through different channels.

Item	What it is
Weights	The learned numbers. With them, users can run the model on their own computers.
Source code	The software to train or run a model
Training data	The examples that set the weights in training

An open weight model releases the weights. It can keep the training data and the training code closed.

How laws use the word

• US, Executive Order 14110 (2023). It defined model weights as “numerical parameters within an AI model that helps determine the model’s output in response to inputs”.
• EU AI Act, Article 53(2). It writes “parameters, including the weights”. Some documentation duties do not apply to a free and open-source release of these parameters. The exception does not apply to models with systemic risks.

Questions and answers

What is the difference between model weights and parameters?

Weights and biases together are the parameters of a model. Executive Order 14110 defined model weights as numerical parameters that help determine the output of a model. The EU AI Act writes "parameters, including the weights".

Does a copy of model weights include the training data and source code?

No. A copy of the weights lets a person run and fine-tune the model. NTIA notes that developers can release weights, code, training data and documentation through different channels. An open-weight model can keep its training data closed.

Why do AI companies keep model weights safe from theft?

A copy of the weights is a full copy of the model. A release cannot be reversed. The ASL-3 standard of Anthropic (May 2025) uses more than 100 security controls for this. An example is a limit on outbound data from the systems that hold the weights.