Acceptance of draft guesses does not fall in long AI reasoning
The large model rejects guesses at reflection words such as wait and however, and not at positions that are far in the text.
Claimed, not confirmed
Researchers measured speculative decoding, a method where a small draft model guesses words and a large model checks them. Some in the industry report that the acceptance rate decreases when reasoning length increases. On all 500 MATH-500 problems, with about 1,780,000 checks, the authors found no decrease in four setups. In the last part of each trajectory, acceptance was a small quantity more.
How it works
The large model was DeepSeek-R1-Distill-Qwen-7B. Two different drafts were used: one from the same family and one from a different family. The authors compared acceptance by position in each problem.
The numbers
Acceptance did not decrease in greedy decoding (0 to 8K positions) or in sampling (0 to 12K). The large model rejects tokens at words such as wait, however and check. A light LoRA training step with three variants did not increase the length of accepted text.
What is not known
The result holds only in the ranges that the authors used, for one large model and one benchmark. The authors first removed a repetition fault in greedy decoding. They do not report if the result holds for other models.
Insight: The industry narrative on reasoning chains may be untested
The authors report that the industry narrative of lower acceptance caused special draft models and training by position. A different October study found that on logical tasks, the answer can be read before COCONUT starts its latent chain. An industry narrative can cause work on a problem that is not real, and this narrative has thin tests. Get a measurement on the output lengths of your system before you make such draft models.
Sources
Posted
