Latency Budgets and Streaming

TL;DR: LLM latency has two distinct parts: time-to-first-token (TTFT), set by prefill (prompt length) and queueing, and inter-token latency, set by decode (tokens-per-second). They feel different: a slow first token feels unresponsive, slow inter-token feels sluggish. Streaming tokens as they are generated hides total latency by showing progress immediately. Designing to a latency budget means allocating time across retrieval, model, and tools, measuring TTFT and tokens/sec (not just end-to-end), and using streaming, caching, and routing to hit the target.

Latency is not a single number

For a normal API, latency is one number. For an LLM, the experience is shaped by two:

• Time-to-first-token (TTFT): how long until the first token appears. Driven by prefill (processing the prompt, which scales with prompt length) plus any queue wait. This dominates perceived responsiveness.
• Inter-token latency: the speed of subsequent tokens (tokens per second), driven by the decode phase (memory-bandwidth-bound, see GPU and serving). This sets how fast the answer streams out.

A system can have great TTFT but slow streaming, or vice versa, and you must measure both, not just end-to-end latency, which hides which phase is the problem.

Streaming hides total latency

Because generation is token-by-token, you can stream tokens to the user as they are produced rather than waiting for the whole response. This transforms the experience: the user sees output start immediately (low perceived latency) even if the full answer takes several seconds. Streaming is one of the highest-leverage UX moves in LLM products, the total time is the same, but perceived latency drops dramatically.

Designing to a budget

Treat latency as a budget allocated across the pipeline:

• Allocate the budget across retrieval, model, and any tool calls; each adds up.
• Cut prefill with shorter prompts and prompt/prefix caching (cached context skips recomputation).
• Cut decode time with smaller/routed models, quantization, and continuous batching, and stream to mask it.
• Cut queueing with enough capacity and good autoscaling.
• For agents, each tool round-trip adds latency, parallelize independent calls.

Why interviewers probe this

Perceived latency makes or breaks LLM UX, and a candidate who reports only one latency number has missed how users experience it. A strong answer separates TTFT (prefill/queue) from inter-token latency (decode), insists on streaming to hide total time, and treats latency as a budget allocated across retrieval/model/tools with concrete levers. That UX-plus-systems framing is exactly the applied-AI sensibility.

Common misconceptions

• "Latency is one number." TTFT and inter-token latency are distinct and feel different; measure both.
• "Streaming reduces total latency." It reduces perceived latency by showing progress; total time is unchanged.
• "Optimize end-to-end latency." You must know whether prefill (TTFT) or decode (inter-token) is the bottleneck to fix the right thing.
• "Latency is purely a serving problem." Retrieval and tool calls also consume the budget; allocate across the whole pipeline.

Key takeaways

• LLM latency splits into time-to-first-token (prefill + queue) and inter-token latency (decode); measure both.
• Streaming shows output immediately, dramatically cutting perceived latency without changing total time.
• Treat latency as a budget across retrieval, model, and tools, with caching, routing, and capacity as levers.
• Diagnose which phase bounds you (prefill vs decode) before optimizing.