ML Infrastructure & GPUs

The interviewer wants real numbers, not 'use a big GPU.' Weights are fixed, the KV cache grows with load, and the lever order decides everything. Here is the back-of-envelope and the serving stack.

The interviewer wants you to know why a model that fits on no single GPU still trains, and to do the optimizer-state memory math that motivates sharding. Here is the parallelism taxonomy and the 16-bytes-per-parameter calculation.

Quantization is the first lever for fitting and speeding up models, and the interviewer wants more than 'use fewer bits.' The signal is knowing what precision buys you, why outliers break naive quantization, and how you prove quality held. Here is that answer.

A favorite at hardware-aware shops. The signal is understanding that attention's cost is memory traffic, not FLOPs, and that FlashAttention is an exact, IO-aware reordering, not an approximation. Here is the answer that shows you think about the memory hierarchy.

The KV cache is why LLM serving is hard, and the interviewer wants the mechanics: what it stores, why it limits concurrency, and the tricks that shrink it. Here is the answer that shows you understand decode-time economics.

Decode is sequential and memory-bound, so latency tricks matter. The signal is explaining speculative decoding's draft-and-verify mechanism (and why it stays exact) plus the other levers and when each applies. Here is the latency toolkit.

Mixed precision is standard at scale, and the interviewer wants the numerics: why FP16 needs loss scaling, why BF16 mostly does not, and what stays in FP32. The signal is understanding dynamic range vs precision. Here is that answer.

A hardware-literacy question, especially at NVIDIA and the labs. The signal is understanding throughput vs latency hardware, the memory hierarchy, and why matrix multiplication maps onto GPUs and TPUs. Here is the architecture-aware answer.

Knowing the algorithms is half the job. The other half is the serving stack that actually delivers throughput inside a latency budget. The signal is matching the server to the workload and naming the four knobs that move the needle.

Distillation buys you a small fast model that keeps most of a big model's quality. The signal is explaining why soft targets transfer more than hard labels, and knowing when to distill versus quantize versus prune. Here is the model-compression answer.

At NVIDIA and the labs this compute-intimacy question is real. The signal is owning the SIMT execution model and the three classic throughput killers, with the concrete fix for each. Here is the low-level answer.

Continuous batching is the single biggest throughput lever in modern LLM serving. The signal is explaining why static batching idles the GPU on variable-length generation, and why PagedAttention is what makes the fix practical.

On-device AI is a real product surface (privacy, offline, latency), and it forces hard constraints. The signal is the quantization plus format plus runtime stack and the tradeoffs you accept under tight memory and battery budgets.

Picking GPUs is a real applied decision, and 'get the biggest one' misses the point. The signal is knowing which spec binds your workload (VRAM, bandwidth, interconnect) and the train-versus-serve difference that flips the answer.

Autoscaling GPUs is not autoscaling web servers: GPUs are scarce and expensive, model loading is slow, and the right signal is not CPU. The signal is scaling on queue and GPU metrics, taming cold starts, and the scale-to-zero economics.

Under load, a shared inference service must decide whose request runs now. The signal is queuing with priorities, backpressure, and fairness wired into batching, not first-come-first-served until the service falls over.

FSDP and ZeRO let you train models too big for one GPU without splitting the compute. The signal is the ZeRO stages (what gets sharded at each level) and the communication you pay for it. Here is the answer.

Distributed training is bottlenecked by GPU-to-GPU communication, and these collectives are how the data moves. The signal is what each one does and which parallelism strategy depends on it. Here is the answer.

Activations, not just weights, can dominate training memory, and gradient checkpointing is the standard fix. The signal is the precise trade: recompute activations in the backward pass instead of storing them. Here is the answer.

When a model is too big for one GPU you split the model itself, not just the data. The signal is distinguishing tensor parallelism (split within a layer) from pipeline parallelism (split across layers) and matching each to the interconnect. Here is the answer.

LLM serving has its own metrics, and one latency number hides the real bottleneck. The signal is splitting prefill from decode and reading GPU utilization as a clue, not a verdict. Here is the diagnostic toolkit.

A PyTorch model in eager mode is not the fastest form for inference. The signal is knowing that ONNX buys portability and TensorRT-style compilers buy hardware-tuned speed, and exactly which optimizations get you there. Here is the answer.

LLM inference has two phases with opposite resource profiles, and co-locating them makes a long prompt stall everyone else's tokens. The signal is knowing why prefill and decode fight, and what separating them costs. Here is the answer.

These four get conflated constantly, but they live at different layers: data processing, distributed training, and general orchestration. The signal is matching the tool to the workload instead of reaching for the one you know. Here is the answer.

Batch size is a training knob with non-obvious effects on speed, memory, and generalization. The signal is the large-batch tradeoffs and the learning-rate-scaling and gradient-accumulation tricks that make scaling actually work.

You rarely write a serving stack from scratch. The signal is knowing what frameworks hand you (batching, multi-model, GPU scheduling) and why LLM-specific servers exist at all when general ones already batch.

OOM is the most common wall in deep learning, and 'buy a bigger GPU' is the weakest answer. The signal is naming the memory consumers, knowing which one dominates, and matching the right lever to it.

nvidia-smi showing 100% can hide that you are using a fraction of the hardware's real compute. The signal is MFU (useful FLOPs vs peak) and the gap between 'the GPU is busy' and 'the GPU is efficient'.

Beyond FP16/BF16, FP8 is the next rung on the precision ladder, and the latest tensor cores compute on it natively. The signal is knowing the two formats, what FP8 actually buys, and why dynamic range forces careful scaling.

At long context the KV cache, not the weights, dominates GPU memory and decode bandwidth. The signal is recognizing it as the binding constraint and naming the levers that shrink it without wrecking quality.

A freshly started replica is slow on its first requests while it loads weights and compiles kernels, and that clashes with reactive autoscaling. The signal is naming the cold-start sources and the warmup and provisioning fixes.

Pruning removes redundant weights to shrink and sometimes speed up a model. The signal is the structured-versus-unstructured split and why only structured pruning reliably wins latency on commodity hardware.

Hosting one full fine-tuned model per customer scales linearly in GPUs and bankrupts you fast. There is a way to put hundreds of variants on one GPU without giving up batching. The interviewer wants to hear how.

CPU-based autoscaling that works fine for a web tier quietly fails on GPU inference: wrong signal, and replicas that take minutes to warm. The interviewer wants the signals you actually scale on and how you hide the cold start.

Paying for accelerators that idle half the time is the most common waste in ML training, and the instinct to add more GPUs makes it strictly worse. The interviewer wants the profiling discipline that finds what is starving them.

The counterintuitive truth of LLM serving: token generation is limited by how fast you can read weights from memory, not by math. Once you see that, the whole optimization menu falls out of one number.

Linear scaling is the marketing number; the real curve bends early for reasons that are physics, not bugs. Here is where the speedup goes and how to claw it back.

A 256-GPU job that wedges with no error at 3am is a special kind of pain. The causes are a short, recurring list. Here is the systematic way to find which one bit you.

Spot GPUs are often 60-90% cheaper, and they vanish with two minutes' warning. The savings are only real if a preemption costs you minutes, not the run. Here is how to make that true.

Naive 4-bit quantization can tank quality, and the reflex to give up and serve fp16 leaves a big speedup on the table. The accuracy is usually recoverable. Here is the ladder of fixes.

Users feel the p99, not the median, and the tail is where serving systems quietly fail. The causes are queuing and batching effects, not a slow model. Here is how to flatten it.

Dedicating a GPU per model strands most of a fleet on idle silicon. Sharing safely is a real systems problem with four distinct mechanisms, each tuned to a different traffic shape. Here is how to pick.

A serving box that's stable on short prompts crashes the moment a 30k-token request lands, because KV-cache memory scales with sequence length times batch. Here is how to bound it without crashing.

At billion-parameter scale, training can be cruising and then the loss jumps and never comes back, burning a fortune in compute. The causes and the playbook are well-known to the few who've done it. Here it is.

Launching an LLM service on a vibes-based GPU count is how you get paged on day one. Capacity planning for LLMs differs from web services because tokens, not requests, are the unit. Here is how to size it.

A single long prompt can freeze every other user's token stream for hundreds of milliseconds. Chunked prefill slices that prompt so decode keeps flowing. Here is the mechanism and the knob that controls it.

A shared system prompt gets re-prefilled on every request unless the server remembers it. Prefix caching skips that work, but only when the blocks line up exactly. Here is the hashing and eviction machinery underneath.

Round-robin routing wastes a feature that doubles throughput: cache locality. Designing the router means choosing a replica by KV state, not just least-loaded. Here is the two-layer design.

Three batching strategies, three very different latency profiles. Picking wrong leaves throughput or tail latency on the floor. Here is what each one costs and when to use it.

GPU memory runs out long before you run out of conversations to cache. Offloading KV to CPU or NVMe buys capacity, but the interconnect can become the new bottleneck. Here is the bandwidth math.

FP8 serving is fast and usually fine, until it silently degrades on the one workload your benchmark did not cover. The signal is knowing what FP8 breaks and how to validate it. Here is the recipe.

Forcing valid JSON sounds free, but the mask computation can stall every decode step. The fix is precompiling the grammar into a fast automaton. Here is where the cost hides and how to remove it.

Most models do not fill a GPU, so one model per GPU burns money. The three sharing mechanisms (MIG, MPS, swapping) have very different isolation and overhead. Here is how to pick.

Autoscaling reacts in minutes, but a burst hits in seconds, and the gap is where your tail latency dies. Absorbing bursts is about buffers and shedding, not just adding replicas. Here is the playbook.

No single parallelism dimension trains a 175B model on its own. The art is layering three of them onto the right hardware so the expensive communication stays on the fast links. Here is how to size each axis.

ZeRO and FSDP both shard training state across data-parallel ranks, but the magic is in when each shard is gathered and freed. Get the memory accounting and the all-gather timing and you can size any run.

Full activation checkpointing saves memory but burns a flat 30% extra compute. Selective recomputation recovers most of that by only recomputing the cheap, memory-heavy operations. Here is how to pick what to recompute.

The collective communication in distributed training is pure overhead unless it runs while the GPU computes. Hiding it is the difference between 30% and 55% MFU. Here is how the overlap actually works and how you confirm it on a trace.

MoE buys you more parameters for the same FLOPs, but the gating network, the load imbalance, and the all-to-all shuffle introduce failure modes dense models never see. Here is the training playbook.

RLHF with PPO is not one model training, it is four models in the loop at once, three of them on the GPU during every step. The memory math and the generation bottleneck are what trip people up.

On a thousand GPUs something fails every few hours. The question is not whether you lose a node, but how many GPU-hours you lose when you do. Checkpoint frequency, sharded writes, and fast restart decide that.

When all-reduce is the bottleneck, the cause is almost always a misconfigured path: traffic on the wrong link, a downed NIC, or a topology NCCL never discovered. Here is the systematic way to find the lost bandwidth.

FP8 promises to roughly double training throughput over BF16, but with only a handful of mantissa bits the numerics are unforgiving. Per-tensor scaling and a selective recipe are what make it converge.

Data, tensor, and pipeline parallelism all leave one sequence's activations on one device, so 200k+ token training hits a wall none of them can fix. The fourth axis shards the sequence itself, and the interview lives in the communication math.

When every request thinks for 30,000 tokens, serving flips from compute-bound prefill to memory-bound decode, and the KV cache becomes the resource you actually schedule. The levers that ruled chat traffic stop being the ones that matter.