TL;DR: A 70B model is ~140GB in FP16, so it does not fit on one 80GB GPU; shard with tensor parallelism across 2+ GPUs, or quantize to INT8/INT4 (~70GB / ~35GB) to fit on one. Then the live constraint is the KV cache, which grows with batch × sequence length and is what actually caps concurrency. Serve with continuous batching and PagedAttention (vLLM), MQA/GQA, and quantization.
How to approach it
Do the arithmetic out loud, separating weights (fixed) from KV cache (grows with load). State the lever order: fit the weights first (quantization or tensor parallelism), then maximize throughput (continuous batching, paged KV cache), then trim tail latency. Most candidates skip the KV cache and quote only weight memory, which is the part that actually limits batch size.
A strong answer
Weights. Parameters × bytes-per-param. FP16/BF16 is 2 bytes, so 70B is about 140GB. That exceeds an 80GB H100/A100, so either tensor-parallel shard across at least 2 GPUs (split each layer's matrices column/row-wise, sync with all-reduce over NVLink) or quantize: INT8 lands near 70GB (fits one 80GB card with room for cache), INT4 near 35GB. Training is a different beast: weights plus gradients plus FP32 Adam states run about 16 bytes/param, roughly 1.1TB, which is why training shards with FSDP/ZeRO. Inference only carries the weights plus the KV cache.
KV cache, the real limiter. Each generated token caches a key and value per layer per head, roughly 2 × layers × hidden × seq_len × batch × bytes. For a 70B model that is on the order of hundreds of KB per token, so a few thousand tokens across a moderate batch is tens of GB, often dwarfing leftover weight headroom. KV cache, not weights, is what caps concurrency.
| Lever | What it buys | When it bites back |
|---|---|---|
| INT8/INT4 quantization | Fit on one GPU, faster decode | Small quality hit at INT4; validate on tasks |
| Tensor parallelism | Fit when one GPU is too small | All-reduce per layer needs NVLink, not PCIe |
| Continuous batching | Biggest throughput win | Larger batch raises time-to-first-token |
| PagedAttention | Kills KV fragmentation, raises batch | Needs a paged-aware kernel (vLLM) |
| MQA/GQA | Shrinks KV cache several-fold | Must be in the model architecture |
Throughput versus latency is the central tradeoff: larger batches raise tokens/sec but raise per-request time-to-first-token, so set the batch policy to the SLO rather than maxing one number.
Key takeaways
- Weights are fixed (2 bytes/param at FP16); the KV cache grows with batch × sequence and is the true concurrency ceiling.
- Lever order: fit weights (quantize or TP), then throughput (continuous batching, PagedAttention), then latency.
- Quantize before reaching for tensor parallelism if a single quantized GPU fits; TP pays interconnect tax.
- Decode is memory-bandwidth-bound, so fewer bytes per weight and a smaller KV cache directly speed generation.
What interviewers probe next
- "Prefill vs decode?" Prefill processes the whole prompt in parallel (compute-bound); decode is one token at a time (memory-bandwidth-bound), which is why KV cache and bandwidth dominate steady-state cost.
- "INT4 quality risk?" Usually a small perplexity hit with AWQ/GPTQ; validate on your eval set, and keep sensitive layers higher precision if a regression shows up.
- "Speculative decoding?" A small draft model proposes tokens the big model verifies in parallel, cutting latency when acceptance is high.
Common mistakes
- Quoting only weight memory and forgetting the KV cache, which is what actually limits batch size.
- Using FP32 numbers for inference; inference runs in FP16/BF16 or lower.
- Reaching for tensor parallelism before quantization when a single quantized GPU would do, paying needless interconnect overhead.
- Conflating training memory (16 bytes/param with optimizer states) with inference memory (weights plus cache).