Time to first token

How time to first token is measured

Time to first token starts when the client sends the request. It stops when the first token of the answer arrives. The NVIDIA benchmark documentation gives three parts:

1. Queue. The request waits for the server.
2. Prefill. The model reads the full prompt and makes the first token.
3. Network. Data goes between client and server.

Prefill time increases with the prompt length. NVIDIA says “longer prompts increase TTFT”. A long document or a full context window gives a longer wait.

TTFT vs the other latency metrics

Metric	What it measures	Most necessary for
TTFT	Wait until the first token	Chat, code completion
Time per output token (TPOT)	Average time between the subsequent tokens	Long answers, summaries
End-to-end latency	TTFT plus the time for all other tokens	Tasks that use the full answer

The DistServe paper (Zhong and colleagues, OSDI 2024) says “real-time chatbots prioritize low TTFT”. Code completion “demands lower TTFT than chatbot”.

Why time to first token matters

TTFT sets how fast an assistant feels. The user sees no text until the first token arrives.

MLPerf Inference, from MLCommons, sets TTFT limits (April 2025):

MLPerf test	TTFT limit	TPOT limit
Llama 2 70B, initial server test	2 s	200 ms
Llama 2 70B, interactive (v5.0)	450 ms, 99th percentile	40 ms, 99th percentile
Llama 3.1 405B (v5.0)	6 s, 99th percentile	175 ms, 99th percentile

MLCommons set these targets from industry research, user surveys, and performance data from platforms such as ChatGPT and Perplexity AI (late 2024).

TTFT is also a security signal. A fast first token can show what is in a shared cache.

TTFT as a timing side channel

A server with a prompt cache does not do the prefill again for a prompt it saw before. The OpenAI documentation says caching cuts “the time spent processing input before the response starts”. The result is a shorter TTFT, and an attacker can measure it.

Gu and colleagues (February 2025) used this signal:

• They set the maximum output to 1 token. Then the full response time is equal to TTFT.
• They audited 17 LLM API providers and found prompt caching in 8.
• In 7 providers, all users shared one cache. A fast answer can show what a different user sent.
• At OpenAI, the shared cache was only for the embedding model text-embedding-3-small. GPT-4o mini shared its cache only in one organization.
• The timing also showed that the OpenAI embedding model is probably a decoder-only transformer.
• They told the providers in October 2024. Five or more providers made changes.

InputSnatch (Zheng and colleagues, November 2024) is a related attack. It uses the time differences that caches cause to steal user input.

Trade-offs

Prefill and decode use the same hardware. DistServe finds that prefill “tends to be compute-bound” and the speed of decode depends on GPU memory bandwidth. One prefill task in a batch of decode tasks slows down the two.

Procedures that make TTFT shorter, and their cost:

• Shorter prompts. Less prefill work, but the model gets less context.
• Prompt caching. The server does not do the prefill again for a cached prefix in the key value cache. A shared cache can leak data.
• Different servers for prefill and decode. DistServe does this. The authors report 7.4 times more requests, or a 12.6 times tighter latency target, in their tests.

Questions and answers

What is the difference between TTFT and time per output token?

TTFT is the wait until the first token of the answer arrives. Time per output token (TPOT), also called inter-token latency, is the average time between the tokens after the first. End-to-end latency is TTFT plus the time for all other tokens. DistServe says chatbots must have a low TTFT, and document summaries a low TPOT.

What is a good time to first token?

No single value is correct for all uses. MLCommons set the initial MLPerf Llama 2 70B server target at 2 seconds TTFT. In April 2025 it added an interactive test with a 99th percentile limit of 450 ms. For the Llama 3.1 405B test, the 99th percentile limit is 6 seconds.

Why does time to first token matter for security?

A short TTFT can show that the server had a prompt in its cache. Gu and colleagues used this signal to find caches shared across users in 7 of 17 LLM API providers. An attacker can use the same signal to learn what other users sent.