Tuning a 4 bit search model can cut its quality by 71.53 points

Researchers did tests on calibration, a small adjustment step, for text embedders that use 4 bit weights. An embedder turns text into numbers for search. On one embedder, the same adjustment range gave results from +1.47 points to -71.53 points of search quality. A rule that does a check of each change on data kept apart accepted no change that decreased the score.

How it works

Quantization makes a model small with less bits for each weight. Calibration adjusts it with a small data set. The distance of the change from the start, the dose, controls the result. The rule accepts a change only if the score is above the score of the model with no tuning. It uses data kept apart.

The numbers

• The same adjustment range gave +1.47 to -71.53 points on EmbeddingGemma.
• The rule accepted 0 changes that decreased the score in 30 decisions.
• It kept 8 of 11 changes that helped.
• In one case the rule did not accept a change that can give 11.13 points more.

What is not known

The authors tell that the rule gives up some result to get safety, and that it is not free. On one model, the results of the 2 methods are not clear. They tell that how many labels are necessary is an open question.

Insight: A check before release matters more than the tuning

A study of 12 October 2026 found that self distillation, a training step, can decrease correct answers. Here, a tuning step can decrease the score by 71.53 points. The 2 results read as proof that a step after training can lower quality with no sign, and a check against a baseline is the control. The groups that hold the check select what goes out.