GraphTok decreases time to first token by 2.1 to 2.4 times with vLLM
One frontend with GraphTok processes 518 requests each second of 121,000-token traffic, against 296 with gigatoken.
Claimed, not confirmed
The researchers made GraphTok, a tokenizer that operates on the GPU. Agent workloads send long contexts again and again, and prefix caching at this time skips most of the prefill. Each request must be tokenized in full before the system can use the cache. The researchers say that this CPU step uses more time than all other parts of the time to first token.
Sources
Posted
