GPU Memory and the Serving Stack
Serving an LLM is mostly a memory problem: the GPU must hold the model weights plus a KV cache that grows with sequence length and batch size, and inference splits into a compute-bound prefill and a memory-bandwidth-bound decode. Knowing the memory math (weights plus KV cache), why decode is bandwidth-bound, and the levers (quantization, batching, paged attention) is the foundation of LLM serving. Applied-AI interviews probe it because 'will this model fit and how fast will it run?' is a constant production question.
TL;DR: LLM serving is dominated by memory. The GPU must hold the model weights (parameters times bytes-per-parameter) plus the KV cache, which grows with sequence length times batch size and often exceeds the weights at scale. Inference has two phases: prefill (process the whole prompt, compute-bound) and decode (generate token by token, memory-bandwidth-bound). Knowing this memory math and which phase bounds you, plus the levers (quantization, batching, paged attention), is the foundation of serving.
The memory math
The first question is "does it fit?" Two consumers:
- Weights: parameters times bytes per parameter. A 70B model is ~140GB in FP16, ~35GB in INT4. This sets the floor.
- KV cache: every token in every active request holds a key and value vector in every layer, so it scales with layers x heads x head-dim x sequence-length x batch. At long context and high concurrency it can exceed the weights and is what caps how many requests you can run at once.
Worked example: does a 70B fit on one A100 80GB?
Take Llama-2-70B (80 layers, hidden 8192, GQA with 8 KV heads, head-dim 128) on a single 80GB A100.
- Weights in FP16: 70e9 params x 2 bytes = 140GB. It does not fit, full stop. Quantize to INT4: 70e9 x 0.5 = 35GB. Now there is ~45GB of headroom for the KV cache.
- KV cache per token: 2 (K and V) x 80 layers x 8 KV heads x 128 head-dim x 2 bytes = 327,680 bytes ≈ 0.33MB/token. (GQA matters here: with 64 query heads instead of 8 KV heads the cache would be 8x larger.)
- Budget the headroom: 45GB / 0.33MB ≈ 136,000 tokens of KV total. That is one 4k-token request times ~34 concurrent users, or a single 128k-context request that nearly eats the entire budget alone.
The takeaway an interviewer wants: weights set whether the model loads at all, but the KV cache sets how many users you can serve concurrently, and at long context the cache, not the weights, is what you run out of.
Prefill vs decode
Inference is two phases with opposite resource profiles:
- Prefill: process the entire prompt in one pass to produce the first token and fill the KV cache. Lots of parallel matrix math, so it is compute-bound, and it drives time-to-first-token.
- Decode: generate tokens one at a time, each reading the whole KV cache. Little compute per step, dominated by streaming weights and KV from memory, so it is memory-bandwidth-bound, and it drives inter-token latency.
This is why throughput is often limited by memory (bandwidth and capacity), not raw compute, and why GPU utilization can look "busy" while you are actually waiting on memory.
| Prefill | Decode | |
|---|---|---|
| Work | whole prompt in one pass | one token at a time |
| Bottleneck | compute (matmul FLOPs) | memory bandwidth |
| Sets | time-to-first-token | inter-token latency |
| Helped by | bigger GPUs, tensor parallelism | quantization, batching, faster HBM |
The serving levers
Because memory is the constraint, the optimizations target it:
- Quantization: fewer bits shrink weights and the KV cache and cut the bandwidth decode must read.
- Continuous batching: serve many requests together to raise throughput (decode is cheap per token; batching amortizes the weight reads).
- PagedAttention: manage the KV cache in pages to avoid fragmentation and fit more concurrent sequences.
- MQA/GQA shrink the KV cache structurally; FlashAttention removes attention's memory-IO overhead.
Specialized serving stacks (vLLM, TGI, TensorRT-LLM) bundle these.
Why interviewers probe this
"Will this model fit on our GPUs and how fast will it run?" is a constant production question, and the answer is memory math plus the prefill/decode distinction. A strong response computes the weight and KV-cache memory, identifies that decode is bandwidth-bound (so throughput is memory-limited), and names the levers (quantization, batching, paged attention). The reserved follow-up is usually "your p99 inter-token latency spikes under load, why?" The strong answer: decode is bandwidth-bound, so as the batch grows the per-step memory traffic grows with it, and once KV-cache capacity is exhausted the scheduler preempts or queues requests, which shows up as tail latency. That is the foundation every other serving optimization builds on.
Common misconceptions
- "Serving is compute-bound." Decode is memory-bandwidth-bound, and capacity (KV cache) often caps concurrency before compute does.
- "Weights are the only memory cost." The KV cache can exceed the weights at long context and high batch.
- "Prefill and decode are the same." Prefill is compute-bound (sets TTFT); decode is bandwidth-bound (sets inter-token latency).
- "High GPU utilization means it is efficient." It can be busy waiting on memory; measure throughput, not just utilization.
Key takeaways
- GPU serving is a memory problem: weights plus a KV cache that grows with sequence length times batch.
- Prefill is compute-bound (time-to-first-token); decode is memory-bandwidth-bound (inter-token latency).
- Throughput is usually limited by memory (bandwidth and KV-cache capacity), not raw compute.
- The levers are quantization, continuous batching, paged attention, and MQA/GQA/FlashAttention.
Check yourself before an interviewer does. Answer from memory first.
Which inference phase is memory-bandwidth-bound?