TL;DR: Concurrency is bounded by KV cache memory, not by compute. A 70B model in fp16 eats about 320 KiB of KV per token, so an 8K conversation costs 2.5 GiB and a 4-GPU node holds roughly 64 of them. That node serves about 3,600 tokens/sec, which covers this demand ten times over. Which is the answer: at 16M tokens/day you would run the fleet at 5 percent utilization and pay roughly 3x the API bill. Size it, then recommend the API until traffic triples.
How to approach it
Do not start with GPUs. Start by asking what number would change the decision, then compute only that. Here it is utilization, and you reach it through four steps: memory per request, requests per node, tokens per second per node, then demand against that capacity. Cost falls out at the end and reverses the premise.
Ask three things before any arithmetic. Output tokens per interaction, because output dominates cost and input dominates memory. Context length, because KV cache scales linearly with it and it is usually the binding constraint. And the peak-to-average ratio, because you buy for peak and pay for all 24 hours.
State your assumptions out loud as you go. An interviewer scoring this is watching whether you know which quantity each assumption controls, not whether you guessed the right constant.
A strong answer
Start with KV cache, since it decides concurrency. For a 70B-class model with 80 layers, 8 key-value heads under grouped-query attention, and head dimension 128, in fp16:
KV per token per layer = 2 (K and V) x 8 heads x 128 dim x 2 bytes = 4 KiB
KV per token = 4 KiB x 80 layers = 320 KiB
KV per 8K conversation = 320 KiB x 8,192 = 2.5 GiB
That 2.5 GiB per active conversation is the number worth memorizing. Now the node. Weights are 70B x 2 bytes = 130 GiB, so a single 80GB card cannot hold the model and two cards leave nothing for cache. Take four:
| Budget on 4x H100 80GB | GiB |
|---|---|
| Total HBM | 320 |
| Model weights (fp16) | 130 |
| Activations, framework, fragmentation | ~20 |
| Left for KV cache | ~170 |
170 GiB divided by 2.5 GiB is about 64 concurrent conversations at full context. Not 640, and not 6. Most candidates never compute this and assume concurrency is a compute question.
Throughput comes from memory bandwidth, not FLOPs, because decoding reads the entire weight set to produce one token. Tensor-parallel across four cards puts 32 GiB of weights on each, and an H100 SXM reads at roughly 3.35 TB/s, so one decode step takes about 10 ms. That is 96 forward passes per second, and each pass emits one token for every sequence in the batch. At 60 percent of theoretical, call it 57 tokens/sec per user and roughly 3,600 tokens/sec for the node.
Now demand. 2,000 employees, 10 interactions a day, 800 output tokens each is 16M output tokens/day. Concentrated in an 8-hour workday that averages 556 tokens/sec, peaking near 1,700 at 3x.
One node covers peak with headroom. Two nodes for redundancy, so eight GPUs, and that is where the estimate turns on the question that prompted it. A node pinned flat out for 24 hours would produce 311M tokens/day. You need 16M. You would run at about 5 percent utilization while paying for every idle hour.
| Monthly | |
|---|---|
| 8 GPUs at ~$2.50/GPU-hour, billed 24/7 | ~$14,400 |
| Same volume at ~$10 per 1M output tokens | ~$4,800 |
| Same volume at ~$3 per 1M output tokens | ~$1,440 |
Self-hosting costs about 3x the API at the high price assumption and 10x at the low one, before anyone is paid to operate it. Break-even against $10/M arrives around 48M tokens/day, roughly 6,000 employees at the same usage. So: the fleet is two nodes if you must build it, and the recommendation is to buy until traffic triples. Say that out loud. The estimate that reverses the plan is the one worth doing.
The honest exceptions, which an interviewer will want you to name unprompted: data residency or an air-gap can make the API unavailable at any price, a fine-tuned model may have no API equivalent, and steady batch workloads (nightly enrichment, document backfill) push utilization high enough to flip the arithmetic, which is why batch and interactive traffic often belong on the same fleet.
What interviewers probe next
"Halve the context to 4K." KV per conversation drops to 1.25 GiB and concurrency roughly doubles to 128. Context length is the cheapest lever you own, which is why prompt discipline is a capacity decision rather than a style one.
"You are memory-bound at 64 users. Now what?" In rough order of return: quantize the KV cache to fp8 for close to 2x concurrency, quantize weights to free HBM for more cache, page the cache so sequences allocate in blocks instead of reserving worst-case (this is what PagedAttention buys), and offload cold sequences. Only then buy cards.
"Where does prefill fit?" Prefill is compute-bound and decode is memory-bound, so they contend badly on one card. A long prompt arriving mid-decode stalls everyone, which is the argument for separating prefill and decode or for chunked prefill.
"What breaks the estimate?" Every assumption is a candidate: a 10x heavier user cohort, agentic traffic that multiplies calls per interaction, or a context window that grows because someone shipped longer retrieval. Name which assumption you would instrument first in production.
Common mistakes
Sizing on FLOPs. Decode is bandwidth-bound, and a compute-based estimate lands far too optimistic.
Forgetting that KV cache scales with concurrency times context, not with users. Two thousand employees are not 2,000 concurrent conversations, and the gap between them is the entire sizing exercise.
Quoting an average and stopping. You buy for peak and pay for 24 hours, so average throughput without a peak factor and a utilization number is not an estimate.
Producing a GPU count and no recommendation. The number is the means. The candidate who computes eight GPUs and then says "do not buy them yet, here is the traffic level where I would" is the one who gets the offer.
Key takeaways
- KV cache per token is
2 x kv_heads x head_dim x bytes x layers. For a 70B in fp16 that is 320 KiB, so 8K context costs 2.5 GiB per conversation. - Concurrency is HBM left over after weights, divided by cache per conversation. Roughly 64 on a 4-GPU node.
- Decode throughput is weight bytes divided by memory bandwidth. Roughly 10 ms per step, 96 steps/sec, one token per sequence per step.
- Utilization decides build-versus-buy, not price per token. Under about 30 percent, the API wins on cost and on the operational headcount nobody budgets.