TL;DR: Profile before you optimize: find whether you are bound by model compute, data and feature fetching, queuing, or network. Then apply no-retrain levers in order: quantization (INT8/FP8) and a compiled runtime (TensorRT, ONNX Runtime), dynamic batching, caching, and only then horizontal replicas. Target p99 specifically, since tail latency is usually queuing and batching effects, not the median.
How to approach it. Insist on measurement first; "add hardware" without a profile is exactly the wrong instinct the question is screening for. Split the latency budget into stages, name the levers cheapest-first, and flag that p99 (tail) has different causes than the median.
A strong answer. Profile. Break end-to-end latency into stages: feature and data fetch, queue wait, model forward pass, post-processing, network. p99 is usually dominated by queuing and batching effects or a slow dependency, not raw model FLOPs, so optimizing the model when the bottleneck is a slow feature store wastes effort.
No-retrain levers, in order:
- Quantization. Cast weights and activations to INT8 or FP8: often a large speedup for a small, validate-able quality hit. Post-training quantization needs no retraining.
- Optimized runtime + graph compilation. Export to TensorRT or ONNX Runtime for kernel fusion, optimized kernels, and reduced overhead, frequently 2x or more with no model change.
- Dynamic batching. Group concurrent requests so the GPU runs efficiently; raises throughput, but cap the batch wait so it does not blow the p99 budget (the classic throughput-vs-latency knob).
- Caching. Cache results for repeated or near-duplicate inputs (and for LLMs, the KV cache and prompt caching); a meaningful fraction of production traffic repeats.
- Distillation or a smaller model (mild retrain) only if the above is insufficient.
- Horizontal replicas + autoscaling. Add capacity to cut queue wait once per-request cost is optimized, otherwise you just pay for inefficiency.
For LLMs specifically: continuous batching and PagedAttention (vLLM), speculative decoding, and serving a quantized model all attack tail latency directly.
| Lever | Cost to apply | Typical win | Watch out for |
|---|---|---|---|
| Quantization | Low | Large speedup | Validate accuracy on eval set |
| Runtime compile | Low | 2x or more | Op coverage, export quirks |
| Dynamic batching | Low | Throughput up | Bounded wait or p99 blows up |
| Caching | Low | Cuts repeat work | Cache key correctness |
| Horizontal scale | High (recurring) | Cuts queue wait | Pays for inefficiency if used first |
The defensible order: measure, optimize per-request cost (quantize, compile, cache, batch), then scale out. Scaling first hides the problem and inflates cost.
Key takeaways
- Profile to find the bottleneck stage before touching the model.
- p99 is a queuing and batching problem more than a FLOPs problem.
- Optimize per-request cost before adding replicas, or you pay for the inefficiency.
- For LLMs, continuous batching, PagedAttention, and speculative decoding target the tail.
What interviewers probe next.
- "Why does batching raise throughput but risk p99?" Waiting to fill a batch adds latency to early-arriving requests; bound the wait or use continuous batching so it adapts to load.
- "Quantization quality risk?" Validate on your eval set; keep sensitive layers higher precision if accuracy drops, and prefer calibrated PTQ methods.
- "GPU utilization is 35%, latency still high, why?" Often CPU-bound preprocessing, data-loading, or small batch sizes starving the GPU; fix the pipeline, not the model.
- "When is horizontal scaling the right first move?" When per-request cost is already low and you are simply out of capacity (queue-bound), not when each call is wasteful.
Common mistakes.
- Adding GPUs or replicas before profiling, paying for inefficiency.
- Optimizing the model when the bottleneck is the feature store or network.
- Cranking batch size for throughput and blowing the p99 latency budget.
- Treating median and p99 as the same problem; tail latency is usually queuing.