Push-based KV transfer in vLLM cuts TTFT by 3 times

The authors added a push-based connector to vLLM, an open-source engine that operates language models. The connector moves the setup of KV cache transfer off the primary path between prefill and decode. The authors report a maximum of 3 times lower time to first token (TTFT) than the default pull mode. They tested one model, Qwen3-32B, and state that results change with the workload.

How it works

The system splits each query. A prefill node reads the prompt. A decode node writes the response. The KV cache is the saved work that moves between them.

In pull mode, the decode node waits until prefill ends, then asks for the data. In push mode, the decode node reserves its target blocks first. The prefill node writes the data when it is prepared. A background thread does the setup.

The numbers

The test used one prefill node and one decode node on Amazon EC2 P5en machines. Push mode gave a maximum of 3 times lower TTFT and 30 percent lower time for each output token. The authors state that the increases are highest at high query rates.

What is not known

The authors give numbers for one model only. They state that results change with the workload and the set-up.

Insight: Rival processor makers now share the same open parts

The connector operates on the NIXL library of NVIDIA, on the AWS network. The authors made it on AWS Trainium processors, then gave it to vLLM for all. The open Nemotron 3.5 Super VL model of NVIDIA came to Nscale on 15 October. Rivals use open parts together, and the competition moves to who sets the defaults in vLLM. Leaders: tell your team to check if your system limits you to one processor.