Google says its TPU Sync library cuts AI answer wait by 90%

Google Cloud released TPU Sync, an open source library for its TPUs, the AI processors that it makes. It moves the KV cache, the work that a model keeps on a text to use again, between TPU memory, host memory and storage. Google says it cuts KV movement latency by more than 90% for open models of 235B to 480B parameters.

How it works

TPU memory is small. When it is full, a model must drop kept context and calculate it again. TPU Sync moves blocks of context to host memory, local disks and storage for all hosts. The moves occur while the TPU continues to operate. It works with PyTorch and JAX, and with engines that give model results.

The numbers

• More than 90% less KV movement latency.
• 90% lower average time to first token, compared with no TPU Sync.
• A maximum of 223 gigabytes each second for each host, from a hardware limit of 256.

What is not known

The numbers are from Google. They compare with a setup that has no TPU Sync, not with a rival. The part for reinforcement learning, a method to train a model, is in test. Google tested it only on internal work.

Insight: Open software with a speed increase for TPUs only

TPU Sync is open source, but it keeps data in the native TPU memory layout to move it fast. Google and Meta also released TorchTPU, and Google released engines for TPUs. This reads as a move: Google releases the software layers that make TPUs fast, but the speed increase is for TPUs only. Monitor if the same tools come for other processors.