serving

The per-token sticker price hides the real decision. Self-hosting only wins past a volume break-even most teams misjudge. Here is the back-of-envelope a staff engineer does on the whiteboard.

Most GPU memory in naive LLM serving is wasted on KV-cache fragmentation. PagedAttention borrows virtual memory paging to reclaim it. The signal is explaining the fragmentation problem and how blocks fix it.

MoE saves compute but is awkward to serve: experts must be sharded, tokens routed across devices, and batches balanced. The signal is the all-to-all communication and the load-imbalance problem, not the training story.

Everyone budgets image tokens. Almost nobody budgets image latency. Images are a prefill problem, prompt caching stops paying for itself, and the cheapest answer is often not to call a VLM at all.

Scarce GPUs, dozens of models, every team wanting low latency at low cost. The signal is whether you can turn that into one governed serving fleet: continuous batching, KV cache, per-tenant quotas, and cost you can actually attribute.

Tagging articles at scale is bread-and-butter ML, but the easy version fails three ways: it is multi-label, the taxonomy is hierarchical and keeps growing, and rare classes hide behind aggregate accuracy. The signal is the model choice plus how you handle all three.

Detecting and boxing objects at scale comes down to one driving tradeoff plus a handful of CV specifics most candidates fumble: the detector family, NMS, focal loss, and why mAP not accuracy is the metric. Here is the design.

GPUs are scarce and expensive, so multi-region HA for LLMs is not just web-app HA with bigger boxes. Capacity, routing, and failover all bend around the GPU constraint. Here is the design.

Classic distributed-systems tradeoffs show up in ML infra with an ML twist: stale features and eventually-consistent reads have model-accuracy consequences, not just correctness ones. Here is how to reason about it.

The interviewer wants real numbers, not 'use a big GPU.' Weights are fixed, the KV cache grows with load, and the lever order decides everything. Here is the back-of-envelope and the serving stack.

The KV cache is why LLM serving is hard, and the interviewer wants the mechanics: what it stores, why it limits concurrency, and the tricks that shrink it. Here is the answer that shows you understand decode-time economics.

Decode is sequential and memory-bound, so latency tricks matter. The signal is explaining speculative decoding's draft-and-verify mechanism (and why it stays exact) plus the other levers and when each applies. Here is the latency toolkit.

Knowing the algorithms is half the job. The other half is the serving stack that actually delivers throughput inside a latency budget. The signal is matching the server to the workload and naming the four knobs that move the needle.

Continuous batching is the single biggest throughput lever in modern LLM serving. The signal is explaining why static batching idles the GPU on variable-length generation, and why PagedAttention is what makes the fix practical.

Under load, a shared inference service must decide whose request runs now. The signal is queuing with priorities, backpressure, and fairness wired into batching, not first-come-first-served until the service falls over.

A PyTorch model in eager mode is not the fastest form for inference. The signal is knowing that ONNX buys portability and TensorRT-style compilers buy hardware-tuned speed, and exactly which optimizations get you there. Here is the answer.

A freshly started replica is slow on its first requests while it loads weights and compiles kernels, and that clashes with reactive autoscaling. The signal is naming the cold-start sources and the warmup and provisioning fixes.

Hosting one full fine-tuned model per customer scales linearly in GPUs and bankrupts you fast. There is a way to put hundreds of variants on one GPU without giving up batching. The interviewer wants to hear how.

Users feel the p99, not the median, and the tail is where serving systems quietly fail. The causes are queuing and batching effects, not a slow model. Here is how to flatten it.

Dedicating a GPU per model strands most of a fleet on idle silicon. Sharing safely is a real systems problem with four distinct mechanisms, each tuned to a different traffic shape. Here is how to pick.

Launching an LLM service on a vibes-based GPU count is how you get paged on day one. Capacity planning for LLMs differs from web services because tokens, not requests, are the unit. Here is how to size it.

A single long prompt can freeze every other user's token stream for hundreds of milliseconds. Chunked prefill slices that prompt so decode keeps flowing. Here is the mechanism and the knob that controls it.

A shared system prompt gets re-prefilled on every request unless the server remembers it. Prefix caching skips that work, but only when the blocks line up exactly. Here is the hashing and eviction machinery underneath.

Round-robin routing wastes a feature that doubles throughput: cache locality. Designing the router means choosing a replica by KV state, not just least-loaded. Here is the two-layer design.

Three batching strategies, three very different latency profiles. Picking wrong leaves throughput or tail latency on the floor. Here is what each one costs and when to use it.

GPU memory runs out long before you run out of conversations to cache. Offloading KV to CPU or NVMe buys capacity, but the interconnect can become the new bottleneck. Here is the bandwidth math.

FP8 serving is fast and usually fine, until it silently degrades on the one workload your benchmark did not cover. The signal is knowing what FP8 breaks and how to validate it. Here is the recipe.

Forcing valid JSON sounds free, but the mask computation can stall every decode step. The fix is precompiling the grammar into a fast automaton. Here is where the cost hides and how to remove it.

Most models do not fill a GPU, so one model per GPU burns money. The three sharing mechanisms (MIG, MPS, swapping) have very different isolation and overhead. Here is how to pick.

Autoscaling reacts in minutes, but a burst hits in seconds, and the gap is where your tail latency dies. Absorbing bursts is about buffers and shedding, not just adding replicas. Here is the playbook.

When every request thinks for 30,000 tokens, serving flips from compute-bound prefill to memory-bound decode, and the KV cache becomes the resource you actually schedule. The levers that ruled chat traffic stop being the ones that matter.