TokenGate cuts the first answer wait by 1.4 to 11.3 times

Stanford and UC Berkeley researchers released TokenGate. It is a module that learns which tokens a large language model can skip in a long input. A token is a small piece of text. The researchers tell that it cuts the wait for the first token of an answer by 1.4 to 11.3 times. It is a drop-in part for a model.

How it works

TokenGate removes the tokens that are less important from a long input. It works in the prefill step. In this step, the model reads the full input before it writes an answer. The researchers tell that it is a drop-in module. They train it.

The numbers

The researchers give 2 ranges. The wait for the first token decreases by 1.4 to 11.3 times. Peak memory decreases by 1.5 to 3.4 times. They tell that accuracy does not decrease.

What is not known

The source names no test, no model and no hardware. It does not tell which conditions give 1.4 times and which give 11.3 times. It gives no test by other teams.

Insight: 2 teams cut the same delay from 2 sides

GraphTok is a tool from other researchers. It speeds up the tokenizing step that comes before the model reads the input. They tell that this step uses more time than all other parts of the wait for the first token. TokenGate works on the next step. 2 groups work on the same wait from 2 ends. It reads as if long inputs for AI agents cost too much now.