The sampler controls if long AI output holds at high temperature
At temperature 2, agents with top-p get 0 solutions for 100 GitHub issues, but agents with top-nsigma get 46 to 62.
Researchers tested 29 open-weight models, from 1B to 122B parameters, on 10 benchmarks. The sampler is the method that selects the next token from the probabilities of the model. The authors tell that it controls if long output holds when temperature increases. Top-nsigma keeps a median 100% of its temperature 1 score at temperature 2, and top-p keeps 26%.
What is new
Output from models gets less good as output length increases, and many teams try to repair this with more training. The paper tells that the sampler is frequently the cause. Samplers that keep a small, stable pool of candidate tokens do not decrease in precision.
The numbers
In one Llama-3.1-8B test, the pool of top-p increases from 4 tokens to 51,845 when temperature goes from 1 to 2. The pool of top-nsigma stays at 3. The number of tokens in the pool and the quality have a Spearman correlation of -0.77.
What is not known
At the default temperature of 1, most samplers do the same. The paper tests open-weight models only. It gives no result for closed models.
Insight: Context length is sold, but the sampler adjustment is free
Vendors offer a large context. LongCat 2.5 has 1M tokens, and GPT 6.1 Sol has a default of 500k in Codex. This paper shows that a free adjustment can control if long output holds. Vendors make the large number the primary point, and the free adjustment gets less notice. Test the sampler before you select a model with more context.
Sources
Posted