Key-value cache

How a KV cache works

A language model writes text one token at a time. Each new token depends on all the tokens before it. The attention mechanism of each layer makes a key and a value for each token.

In a decoder transformer, the keys and values of earlier tokens do not change. Thus the model keeps them and does not calculate them again.

For each step	Without a cache	With a cache
Keys and values calculated	All tokens, again	The new token only
Attention cost for each step	Quadratic with length	Linear with length
Memory for keys and values	Not kept	Grows with each token

Source: Hugging Face Transformers documentation. The cache is for inference only, not for training.

Why the KV cache matters

After the weights, the KV cache can use the most GPU memory at inference. The vLLM paper (Kwon et al., September 2023) gives these numbers:

• 800 KB: the cache of one token of OPT-13B.
• 1.6 GB: the maximum cache of one request, at 2,048 tokens.
• About 30%: the GPU memory for request data, for a 13B model on an NVIDIA A100 40GB. The weights used about 65%.
• 20.4% to 38.2%: the part of cache memory that older systems used for token data. The remaining memory was waste.

Memory sets how many requests a server can operate at the same time. vLLM manages the cache in pages, as an operating system does. The paper reports 2 to 4 times the throughput at the same latency.

The cache is also a security surface. When two users share a cache, the response time can show what the other user sent.

In practice

Prompt caching keeps the cache between requests

Providers keep the KV cache of a prompt prefix for subsequent requests. OpenAI says its prompt cache “stores key-value (KV) tensors, not the tokens themselves”. A long system prompt in each request is the usual use.

Provider (documentation, read November 2026)	Cached input price	Default lifetime	Sharing
OpenAI	Discount of a maximum of 95%	Default 24h retention: about 30 minutes, maximum 24 hours. In-memory retention: about 5 to 10 minutes without use, maximum 1 hour	Not across organizations
Anthropic	0.1 times the usual input price, with some model differences. Write: 1.25 times	5 minutes, or 1 hour at more cost	Not across organizations

Three leaks from a shared cache

1. Same prefix. A cached prompt is faster to process. Thus the timing changes with the data. Gu and colleagues say an attacker “could, in principle” find that a different user sent the same prefix.
2. Prompt recovery. Wu and colleagues (NDSS 2025) show PromptPeek. The attack uses KV cache sharing in SGLang and vLLM to reconstruct the prompts of other users.
3. Model data. Gu and colleagues used timing to find that the OpenAI embedding model is a decoder-only Transformer. This is a type of side channel model extraction.

Gu and colleagues did their audits in September and October 2024. They found sharing across users at seven API providers, with OpenAI. After disclosure in October 2024, five or more providers made changes.

How to stop cross-user leaks

• Do not share a cache across tenants. OpenAI and Anthropic isolate caches for each organization.
• On a self-hosted vLLM server, give each tenant a cache salt. Only requests with the same salt use the same cached blocks.
• Do not put a secret in a shared prompt prefix. A cache hit can show that the prefix exists.

Trade-offs

A smaller cache saves memory, but it can decrease quality.

• Multi-query attention (Shazeer, 2019) shares one set of keys and values across all attention heads. Ainslie and colleagues say it “drastically speeds up decoder inference” but can cause “quality degradation”.
• Grouped-query attention (Ainslie et al., 2023) uses a small number of key and value heads. The authors report quality near full multi-head attention, at a speed near multi-query attention.
• Shared prefix caches decrease cost and wait time. But they add a timing side channel if tenants do not have isolated caches.

Questions and answers

What is the difference between a KV cache and prompt caching?

A KV cache keeps the keys and values of earlier tokens during one response. Prompt caching keeps that cache after the response. A subsequent request with the same prefix can then use it. OpenAI says its prompt cache stores key-value tensors, not the tokens.

Can a KV cache leak data between users?

Yes, when users share it. A cached prefix gives a faster response, and an attacker can measure that time. Gu and colleagues found cross-user sharing at seven API providers in 2024. OpenAI and Anthropic say that they do not share prompt caches between organizations.

Why does a KV cache use so much memory?

The cache keeps a key and a value for each token, in each layer. In the vLLM paper, one token of OPT-13B used 800 KB. One request used a maximum of 1.6 GB.