Model precision (FP16, BF16, 8-bit, 4-bit)

What each format holds

A floating point number has a sign bit, exponent bits for range, and fraction bits for precision.

Format	Bits	Layout or method
FP32	32	single precision
FP16	16	1 sign, 5 exponent, 10 fraction
BF16	16	1 sign, 8 exponent, 7 fraction
FP8	8	E4M3 or E5M2
INT8	8	integer with a scale
4-bit	4 to 4.5	NF4, or Q4_K in GGUF

Sources: NVIDIA, Google (2019), Micikevicius et al. (2022), Dettmers et al. (2022, 2023), Hugging Face.

How FP16, BF16 and low-bit formats are different

• FP16 has more fraction bits. Thus its values are more accurate. Its largest normal value is 65,504 (NVIDIA). FP16 training frequently uses loss scaling. Without it, small gradients can become zero.
• BF16 has the range of FP32 (Google, 2019). Google says BF16 can replace FP32 with almost no changes.
• 8-bit and 4-bit come from quantization of a 16-bit model. LLM.int8() keeps some outlier values in 16-bit and does “more than 99.9%” of the math in 8-bit.

Why model precision is important

Precision sets the memory of a model. For the weights, memory is parameters times bytes for each weight.

BLOOM-176B weights	Memory
BF16	352 GB (Hugging Face, Aug 2022)
8-bit	about 176 GB
4-bit	about 88 GB

The 8-bit and 4-bit rows are our arithmetic.

For security teams, the precision is part of the deployed model. Egashira and colleagues (2024) showed an attack. They made a model that seemed benign in full precision. When quantized with some methods, it wrote vulnerable code.

In practice

Three results show what lower precision gives:

1. LLM.int8() operated models of a maximum of 175B parameters in 8-bit “without any performance degradation” in the tests of the authors (2022).
2. QLoRA fine-tuned a 65B model on one 48 GB GPU. It kept “full 16-bit finetuning task performance” (2023).
3. Meta changed Llama 3.1 405B from BF16 to FP8 to fit it in “a single server node” (July 2024).

A GGUF file names its precision type, for example F16, BF16, Q8_0 or Q4_K. Use these steps when you select or examine a model:

• Record the precision of each model in your inventory, with the source of the file.
• Test the precision you deploy. A test of the BF16 source model does not include the Q4_K copy.
• Get quantized files from a known source, or quantize a verified copy of the source model.
• Start memory plans at 2 bytes for each weight at 16-bit. Add memory for the context cache.

Questions and answers

What is the difference between FP16 and BF16?

The two formats use 16 bits for each weight. FP16 has 5 exponent bits and 10 fraction bits. Its values are more accurate, but its largest normal value is 65,504. BF16 has 8 exponent bits and 7 fraction bits. It has the same range as FP32, with less accurate values. Google says BF16 usually trains with no loss scaling.

How much memory does a model use at each precision?

For the weights, multiply the number of parameters by the bytes for each weight. Use 4 for FP32, 2 for FP16 or BF16, 1 for 8-bit and about 0.5 for 4-bit. Hugging Face gives BLOOM-176B as 352 GB in BF16. The context cache and training data use more memory.

Can an attacker hide bad behavior in a quantized model?

Yes, in a research attack. Egashira and colleagues (2024) made a model that was benign in full precision. It was malicious when quantized with some methods. Thus a test of the full precision model did not show the behavior. Test the precision file that you deploy.