vLLM adds two-way cache transfer that cuts agent wait by 3.3 times

The authors gave vLLM a new tool. vLLM is an open-source engine that operates language models. The tool sends the key-value (KV) cache back from a decode node to a prefill node on the next turn of a chat. The KV cache is the saved work of a model on a text. The authors state that it cuts the time to first token (TTFT) by a maximum of 3.3 times.

How it works

The system splits each query. A prefill node reads the prompt. A decode node writes the response.

In the usual set-up, the saved work moves only from prefill to decode. Agents send the full history on each turn, and the prefill node does the same work again. Now the decode node keeps its blocks for 480 seconds by default, and a router tells the prefill node where they are.

The numbers

The tests used one prefill node and one decode node on Amazon EC2 P5en machines. TTFT fell by a maximum of 1.6 times for Qwen3-32B and 3.3 times for GLM-5-FP8. The authors state that results change with the workload and the set-up.

What is not known

The 480 second hold is set, and the router does not extend it. A turn after that uses the usual path. No test of bigger set-ups is reported.

Insight: Saved model work is becoming the item that systems trade

SwaT, reported on 14 October, sends KV cache between models of different families, at 4 times the speed of text. This tool and SwaT use saved work, not text, as what moves, which seems to make the cache the area where cost is won. The authors made this on AWS Trainium processors and gave it to vLLM, maybe to make that processor a usual alternative.