inference

MoE is why some frontier models have huge parameter counts but serve cheaply. The signal is the sparse-activation idea, the routing mechanism, and the operational cost (memory, load balancing) that the FLOP savings quietly hide.

Neural text degeneration is a known failure of likelihood-maximizing decoding, not a broken model. The fixes are specific decode-time knobs. Here is which lever to pull and what each one costs.

The per-token sticker price hides the real decision. Self-hosting only wins past a volume break-even most teams misjudge. Here is the back-of-envelope a staff engineer does on the whiteboard.

MQA and GQA shrink the KV cache by sharing key/value heads. MLA takes a different route: compress K and V into a low-rank latent and cache that. The signal is knowing it is a cache trick, not a head-sharing trick, and why it keeps quality.

Drop the oldest tokens to bound the KV cache and the model collapses. Keep just the first few tokens and it generates indefinitely. The signal is understanding why those first tokens act as an attention sink.

At long context the KV cache, not the weights, fills the GPU. The signal is naming the levers (quantization, token eviction, head sharing, low-rank) and what each one costs in quality.

Most GPU memory in naive LLM serving is wasted on KV-cache fragmentation. PagedAttention borrows virtual memory paging to reclaim it. The signal is explaining the fragmentation problem and how blocks fix it.

Speculative decoding speeds up generation without changing the output distribution. The signal is knowing how draft-model, self-drafting (Medusa/EAGLE), and lookahead approaches differ in where the guesses come from.

Both squeeze an LLM to 4-bit weights, but they decide what to protect very differently. The signal is GPTQ's error-correcting solve versus AWQ's activation-aware scaling, and the calibration each needs.

GGUF is the format behind most local LLM runs, and its quant names look cryptic. The signal is decoding what Q4_K_M means and why mixed-precision k-quants beat naive uniform quantization.

FP8 is the format behind modern H100-class training and serving. The signal is knowing the two FP8 variants, why a floating format beats INT8 for dynamic range, and where scaling still matters.

MoE saves compute but is awkward to serve: experts must be sharded, tokens routed across devices, and batches balanced. The signal is the all-to-all communication and the load-imbalance problem, not the training story.

Grammar-constrained decoding sounds like a solved problem until tokenization misaligns with the grammar. The signal is the token-vs-character boundary mismatch and how prefix automata and token healing fix it.

Temperature 0 does not mean deterministic, and the reason is in the GPU kernels, not the sampler. The signal is naming the batch-invariance problem and knowing which fixes are real versus placebo.

Implementing the KV cache proves you understand why decode is efficient: you append the new token's K/V and reuse the rest instead of recomputing. The signal is the append-and-attend-over-cache logic. Here is the implementation.

Scarce GPUs, dozens of models, every team wanting low latency at low cost. The signal is whether you can turn that into one governed serving fleet: continuous batching, KV cache, per-tenant quotas, and cost you can actually attribute.

The trap is jumping straight to 'add more GPUs.' The signal is profiling first, then applying the cheap, no-retrain levers in the right order. Here is the diagnosis-then-optimize playbook for p99.

GPU spend is usually the biggest line item in an AI org, and this question checks whether you think in utilization, not just capacity. The signal is squeezing per-unit cost (utilization, right-sizing, spot, quantization) before scaling out.

Not every prediction needs a low-latency endpoint, and defaulting to one burns money. The signal is matching the serving pattern to the latency and freshness requirement. Here is the decision and when you'd reverse it.

A Deployment and a Service will serve a model, but GPUs break every Kubernetes default: scheduling, probes, autoscaling signals, and rollouts. Here is the setup that survives production, and when KServe earns its complexity.

The interviewer wants real numbers, not 'use a big GPU.' Weights are fixed, the KV cache grows with load, and the lever order decides everything. Here is the back-of-envelope and the serving stack.

Quantization is the first lever for fitting and speeding up models, and the interviewer wants more than 'use fewer bits.' The signal is knowing what precision buys you, why outliers break naive quantization, and how you prove quality held. Here is that answer.

The KV cache is why LLM serving is hard, and the interviewer wants the mechanics: what it stores, why it limits concurrency, and the tricks that shrink it. Here is the answer that shows you understand decode-time economics.

Decode is sequential and memory-bound, so latency tricks matter. The signal is explaining speculative decoding's draft-and-verify mechanism (and why it stays exact) plus the other levers and when each applies. Here is the latency toolkit.

Distillation buys you a small fast model that keeps most of a big model's quality. The signal is explaining why soft targets transfer more than hard labels, and knowing when to distill versus quantize versus prune. Here is the model-compression answer.

Continuous batching is the single biggest throughput lever in modern LLM serving. The signal is explaining why static batching idles the GPU on variable-length generation, and why PagedAttention is what makes the fix practical.

Autoscaling GPUs is not autoscaling web servers: GPUs are scarce and expensive, model loading is slow, and the right signal is not CPU. The signal is scaling on queue and GPU metrics, taming cold starts, and the scale-to-zero economics.

LLM serving has its own metrics, and one latency number hides the real bottleneck. The signal is splitting prefill from decode and reading GPU utilization as a clue, not a verdict. Here is the diagnostic toolkit.

LLM inference has two phases with opposite resource profiles, and co-locating them makes a long prompt stall everyone else's tokens. The signal is knowing why prefill and decode fight, and what separating them costs. Here is the answer.

You rarely write a serving stack from scratch. The signal is knowing what frameworks hand you (batching, multi-model, GPU scheduling) and why LLM-specific servers exist at all when general ones already batch.

At long context the KV cache, not the weights, dominates GPU memory and decode bandwidth. The signal is recognizing it as the binding constraint and naming the levers that shrink it without wrecking quality.

The counterintuitive truth of LLM serving: token generation is limited by how fast you can read weights from memory, not by math. Once you see that, the whole optimization menu falls out of one number.

Naive 4-bit quantization can tank quality, and the reflex to give up and serve fp16 leaves a big speedup on the table. The accuracy is usually recoverable. Here is the ladder of fixes.

Users feel the p99, not the median, and the tail is where serving systems quietly fail. The causes are queuing and batching effects, not a slow model. Here is how to flatten it.

A serving box that's stable on short prompts crashes the moment a 30k-token request lands, because KV-cache memory scales with sequence length times batch. Here is how to bound it without crashing.

Launching an LLM service on a vibes-based GPU count is how you get paged on day one. Capacity planning for LLMs differs from web services because tokens, not requests, are the unit. Here is how to size it.

A single long prompt can freeze every other user's token stream for hundreds of milliseconds. Chunked prefill slices that prompt so decode keeps flowing. Here is the mechanism and the knob that controls it.

A shared system prompt gets re-prefilled on every request unless the server remembers it. Prefix caching skips that work, but only when the blocks line up exactly. Here is the hashing and eviction machinery underneath.

Three batching strategies, three very different latency profiles. Picking wrong leaves throughput or tail latency on the floor. Here is what each one costs and when to use it.

Forcing valid JSON sounds free, but the mask computation can stall every decode step. The fix is precompiling the grammar into a fast automaton. Here is where the cost hides and how to remove it.

When every request thinks for 30,000 tokens, serving flips from compute-bound prefill to memory-bound decode, and the KV cache becomes the resource you actually schedule. The levers that ruled chat traffic stop being the ones that matter.