Small language model

How it works

A small language model works like all other language model types. It predicts the next token from the text before it. The difference is the number of parameters, and thus the hardware it uses.

Makers use three usual methods to make a model small:

1. Pruning. Remove parts of a larger model. Meta cut Llama 3.2 1B and 3B from Llama 3.1 8B with “structured pruning”.
2. Distillation. Train the small model on the outputs of a larger one. Meta used “logits from the Llama 3.1 8B and 70B models”.
3. Quantization. Keep each weight in fewer bits. The Apple on-device model uses an average of 3.7 bits for each weight.

Microsoft trained the 3.8 billion parameter Phi-3-mini on 3.3 trillion tokens. It got 69% on the MMLU benchmark (Microsoft, April 2024).

What a small model changes

A small model can operate where a large model cannot. That changes who holds the data, and the energy that each answer uses.

Model	Parameters	Device	Speed	Source, date
Phi-3-mini, 4-bit	3.8B, about 1.8 GB	iPhone 14, fully offline	more than 12 tokens/s	Microsoft, 2024
Apple on-device model	about 3B	iPhone 15 Pro	30 tokens/s	Apple, June 2024
Llama 3.2 1B and 3B	1B and 3B	edge and mobile devices	not given	Meta, Sep 2024

For security teams, three points follow:

• Data stays local. Meta says on-device use gives “strong privacy where data never leaves the device”.
• Less latency and energy. NVIDIA researchers say a 7B model is 10 to 30 times cheaper in latency, energy and FLOPs than a 70B to 175B model (June 2025). They add that the economics are “highly case-specific”.
• The model is on the endpoint. The weights and the runtime are assets that you must keep safe. See security risks of running LLMs locally.

Trade-offs

A small model knows less. The Phi-3 authors write that the model “does not have the capacity to store too much factual knowledge”. They propose a search engine to fill the hole. That is the method of retrieval augmented generation.

Other limits:

• Narrow knowledge. IBM says SLMs “lack the wide knowledge base” of larger models.
• Bias from the teacher. IBM also says a small model can learn the bias of the larger model that taught it.
• Unclear savings. The NVIDIA authors write that the economics are “highly case-specific”.

NVIDIA researchers argue that small models are “sufficiently powerful” and more suitable for many tasks in an AI agent system. For general conversation, they prefer systems that use more than one model. Their paper also gives the other view: “LLM generalists will always have the advantage of more general language understanding.”

Where definitions disagree

The sources do not agree on where “small” stops.

• By device. NVIDIA: a model that fits “onto a common consumer electronic device” and answers one user fast.
• By count, below 10 billion. NVIDIA also counts “most models below 10bn parameters” as small.
• By count, some billion at most. IBM: “a few million to a few billion”.
• By the name a maker gives. Microsoft uses “small language model” for Phi-4, at 14 billion parameters.

When a vendor says “small”, check the parameter count and the target hardware, not only the name.

Questions and answers

How many parameters does a small language model have?

No standard sets a number. NVIDIA researchers count most models below 10 billion parameters as small (June 2025). IBM gives a range of a few million to a few billion. Microsoft uses the term for its 14 billion parameter Phi-4.

Is a small language model safer than a large one?

Not by itself. A small model on the device can keep prompts and data on the device. Meta gives this as a privacy benefit of Llama 3.2 1B and 3B. But the model holds fewer facts. IBM says it can also learn the bias of a larger model. Check its outputs as you check the outputs of a large model.