Google releases XLA Serving Engine, a C++ runtime for AI models

Google released XLA Serving Engine (XSE) as open source software. It is a runtime, the software that operates a model, for teams that make models in Python but must use C++. Google tells that XSE operates more than 6 million queries each second in video and ad services of Google. Google gives it to external users.

How it works

Engineers export a model from Python as an XLA SavedModel. A C++ program then reads it and operates the model. This removes the Python interpreter, the program that reads Python code, and the work that it adds. Google tells that XSE is for models that are not, for the most part, large language models.

The numbers

For short requests, Google gives these results from tests by Google:

• 2 to 5 times the throughput of the best Python server that Google could make.
• 8 to 36 times the throughput of a basic Python server.
• 30 to 70 percent less CPU time for each prediction.
• A quarter of the memory.

What is not known

No external team has given results. The results are for short requests, and Google gives no result for long requests.

Insight: Google makes a route to its TPU chips for each type of model

On 21 October 2026 Google released vLLM and SGLang engines for its TPU chips, for large language models. It also added a C++ API to LiteRT for models on devices. XSE includes other models in C++ servers. This reads as a step to include each type of model, so no code path stops a move to TPU. Teams that select chips must check which software work is necessary for a move.