Open-weight model

What is released and what is not

A model’s weights are the numbers it learned in training. Together with its architecture, they help determine what it outputs for a given input. The US National Telecommunications and Information Administration (NTIA) calls a large foundation model open when its weights are released to the public.

An API-only model keeps its weights private: users send inputs and get outputs, but never hold the model. NTIA lists giving weights only to vetted researchers as a narrower option.

Open weights is also short of open source. The Open Source Initiative’s definition adds the training code and enough information about the training data for a skilled person to build an equivalent system.

Why it matters for security

Holding the weights means holding the controls. NTIA says adversarial actors “can remove safeguards from open models via fine-tuning, then freely distribute the model”. Arditi and colleagues found refusal in 13 open chat models, up to 72B parameters, sat in a single direction inside the model. Editing that direction out of the weights made the models far less likely to refuse harmful requests, a jailbreak the authors estimate costs under $5 of compute for a 70B model. They note it may not generalise to models they did not test. See abliterated vs uncensored models.

The release is permanent. Per NTIA, developers “cannot rescind access to the weights or perform moderation on model usage”, and a file pulled from one site can still spread by other means.

So a refusal trained into an open-weight model is a weaker barrier than on an API-only model. Users of a hosted copy can try to prompt around it, as with closed models. Anyone who holds the file can edit it out.

The same property is a benefit. Running a model yourself means your data never reaches the developer, which NTIA says can matter for confidentiality.

Where definitions disagree

• NTIA (2024) asks whether a large foundation model’s weights were released openly to the public. The licence does not enter the definition.
• The Open Source Initiative defines open weights as the final weights and biases of a trained network, and says they “differ significantly from Open Source AI” because the training code and data are missing.
• The EU AI Act exempts a model from two documentation duties when it is under a free and open-source licence and its weights, architecture and usage information are public (Article 53(2)). Monetised releases do not get it (Recital 103), and neither does any general-purpose AI model with systemic risk.

So a large model whose weights anyone can download under a licence that restricts use counts as open to NTIA, but may not meet the Act’s exemption, which asks for a licence allowing access, use, modification and distribution.

Questions and answers

Is an open-weight model the same as an open-source model?

No. An open-weight model publishes its trained weights. The Open Source Initiative's Open Source AI Definition also requires the complete code used to train and run the model and enough information about the training data for a skilled person to build a substantially equivalent system. A model can publish its weights without either.

Can a developer take back an open-weight model after release?

Not in practice. The US NTIA's 2024 report says developers who release weights publicly cannot rescind access to them. The files can be removed from a platform such as Hugging Face, but anyone who already downloaded them can share them by other means.

Do safety refusals protect an open-weight model from misuse?

Not from whoever holds the weights, because they can change them. Arditi et al. (2024) made 13 open chat models far less likely to refuse harmful requests by editing their weights, and estimate the edit costs under $5 of compute for a 70B-parameter model. The US NTIA notes that safeguards can be removed by fine-tuning and the result freely redistributed.