Parameters

How a model gets its parameters: 2 steps

1. Training. Gradient descent adjusts the weights and biases to decrease loss, the measure of model error.
2. Inference. The model uses the learned weights to make predictions.

How to count the parameters of one layer

A count is arithmetic on the shape of each layer. A linear layer in PyTorch has one weight matrix and one bias vector. We calculated this example for a layer with 1,000 inputs and 1,000 outputs:

Part	Shape	Parameters
Weights	1,000 x 1,000	1,000,000
Bias	1,000	1,000
Total		1,001,000

What a parameter count tells you: 3 questions

Read a parameter count as a measure of how large a language model is, not as a score.

Question	What the count tells you	Source example
How much memory?	The total count sets it	Muse Spark Open keeps all 256 experts in memory
How much compute for each token?	The active count sets it	Muse Spark Open runs at the speed of a 42B model
How good is the model?	The count does not tell you	Chinchilla (70B) outperformed Gopher (280B)

In practice

Published counts, 2020 to 2026

These counts come from the paper or the model card of each model. They are the claims of the authors. “Not given” means that the source of this page gives no active count.

Model	Date	Total	Active for each token
GPT-3	May 2020	175B	Not given
Gopher	Compared in March 2022	280B	Not given
Chinchilla	March 2022	70B	Not given
Llama 2, largest released	July 2023	70B	Not given
Mixtral 8x7B	January 2024	47B	13B
Muse Spark Open	October 2026	816B	42B

3 numbers to read on a model card

Use Muse Spark Open as the example, an open weight model from Meta, released on 15 October 2026:

• Total: 816B. This mixture of experts model holds 256 experts, and all of them stay in memory.
• Active: 42B. The model uses 8 experts for each token, which is about 42B parameters.
• Hub count: 822B. The header of the Hugging Face page shows this number. The card does not tell why it is different from 816B. Write which count you use.

From a count to a memory estimate

Model precision gives the bytes for each parameter. BF16 uses 16 bits, which is 2 bytes.

• We calculated: 816B x 2 bytes = about 1.6 TB for the weights in BF16.
• The Meta model card: a GPU with a minimum of 2TB of VRAM for BF16, and 700GB for the NVFP4 version.

Use the value on the model card when you buy hardware. The calculated value is only a lower limit.

Parameters vs hyperparameters

	Parameters	Hyperparameters
Who sets them	The model learns them in training	A person or a tuning service
Examples	Weights, biases	Learning rate, iterations
When they change	During training	During successive training runs

Parameters behind an API are not fully hidden

Model extraction attacks can get some parameters through usual API access. In 2024, Carlini and colleagues got the embedding projection layer of production language models:

• OpenAI Ada and Babbage: the full projection matrix, for less than $20.
• gpt-3.5-turbo: the hidden dimension size, and an estimate of less than $2,000 in queries for the full matrix.

The attack got one layer, not all the parameters of a model.

Trade-offs

More parameters use more memory, and they do not always give more quality. Hoffmann and colleagues trained more than 400 language models. Their Chinchilla model used the same compute budget as Gopher, with a quarter of the parameters, and it outperformed Gopher.

That result compares two models on one budget. It does not show that a smaller model is always better.

Questions and answers

What does 816B parameters mean?

The phrase 816B parameters means that a model holds about 816 billion learned numbers. Hugging Face writes "approximately 816 billion parameters" for Muse Spark Open, a Meta model of October 2026. The B is not bytes. In BF16, each parameter uses 2 bytes. We calculated about 1.6 TB for the weights.

What is the difference between a parameter and a hyperparameter?

A model learns its parameters in training. A person or a tuning service sets hyperparameters, such as learning rate, and adjusts them during successive training runs.

Does a model with more parameters give better results?

Not always. Chinchilla, with 70B parameters, outperformed Gopher, with 280B, on the same compute budget in a 2022 study. That result compares two models on one budget. It does not show that a smaller model is always better.