Google releases vLLM and SGLang engines for its TPU chips

Google released two engines that operate AI models on its TPU chips: vLLM and SGLang. The two engines use PyTorch tools through a new layer, TorchTPU. Google made them with Inferact and RadixArk, the firms of the engine creators. Google reports that preview customers use them in production, and Meta operates SGLang on TPU today.

How it works

TorchTPU makes the TPU a usual PyTorch device, and developers keep their tools. Each engine keeps its scheduler and cache manager and adds TPU code below them. A new open-source library, TPU Sync, moves the KV cache (the work on a text that a model keeps for use again) between machines with no copy step on the host.

The numbers

Google reports 2.01 cents for each million tokens for Qwen 3.5 on Ironwood TPU, on the open InferenceX benchmark. A Kimi K3 model with 2.8T parameters gave more than 700 tokens each second to one user. The Kimi K3 result on AgentX, a test of agent work, is a first result that can change.

Not known

The text does not name the other chips that Google compared. Only preview customers use the engines. The only report of use in production is from Zongwei Zhou of Meta.

Insight: Agent speed depends on how you move saved work

The Google engines move the KV cache between hosts with TPU Sync. A vLLM tool sends the cache back for the next turn. In a test on Amazon GPUs, it cut the time to first token by a maximum of 3.3 times. The two moves read as a push to make TPU easy for PyTorch users and to cut the cost of the cache. When you select hardware, test cache movement, not only chip speed.