Valkey 9.2 adds an AI cache that claims a maximum of 40% lower cost
Multiverse Computing says it served 50% more requests on the same GPUs (processors for AI work) with the new cache.
Valkey is an open source datastore that keeps data in memory, and the Linux Foundation governs it. Version 9.2 adds a shared open source cache for AI inference, the work of a model that gives a reply. The project claims 40% lower inference cost and 80% less wait for the start of a reply, as maximum values.
How it works
A shared cache keeps the work of a model on input text, and many servers use that work again. Teams connect it to the engines vLLM and SGLang through LMCache with one adjustment. A new data type, Path Hash, finds the input that matches the most, with one command.
The numbers
Multiverse Computing says it served 50% more requests on the same GPUs and kept its one second median wait target. It says the median wait for a reply fell 23% at usual traffic and 95% at peak. The project also claims 50% to 65% less replication traffic between regions.
What is not known
The 40% and 80% values are maximums. The source gives one customer result, from the project release, and no other production test.
Insight: Rival clouds share one cache layer for AI
Other tools for agents cut the same work again and again in one engine. Valkey moves the work into a shared layer that AWS and Alibaba Cloud help make. This reads as a race to set the cache standard, since the layer that most teams use sets the standard. Find how much of each turn is the same work again before you buy more GPUs.
Sources
- Valkey shipping forecast9.2 arrives, the CVE tsunami gets a drill, and AI inference bills take a hit
Posted