llm-d 1.0 is a stable open-source system to operate AI models

The llm-d project released version 1.0 at the PyTorch Conference North America. llm-d is open-source software that operates AI models on many servers. Red Hat, IBM and Google keep it. The new version adds a stable API, teams with more important and less important work can use the same servers, and upgrades with no stop. In one test, its router found 97% of the cache that it could use again. The cache is the kept work of the model.

How it works

llm-d operates on Kubernetes, a tool that controls groups of servers. It works with the engines vLLM and SGLang. It sends each query to a server that holds the kept work of the model for that query. Then the model does not do the same work again. Flow control stops a large group of agent queries. Then a person does not wait.

The numbers

IBM and Red Hat give this result. In a test, 3,000 agents that do software work operated at the same time on servers of the team. The cost was 5 to 10 times less than the price of the same open model from a cloud provider.

What is not known

No other group did a test of these numbers, and the source gives no test data. The methods in llm-d are not new. The new part is that they operate together in one stack.

Insight: Many groups build the open agent stack at the same time

NVIDIA, a co-founder of llm-d, released its GPU platform for AI work as open-source software on 13 October 2026. On 19 October 2026, a vLLM tool cut agent wait by a maximum of 3.3 times. Many groups release open tools for agents in the same week, and it looks like each wants its tool to be the standard. Select parts that work with vLLM and Kubernetes, to keep your alternatives open.