gpu

The per-token sticker price hides the real decision. Self-hosting only wins past a volume break-even most teams misjudge. Here is the back-of-envelope a staff engineer does on the whiteboard.

Scarce GPUs, dozens of models, every team wanting low latency at low cost. The signal is whether you can turn that into one governed serving fleet: continuous batching, KV cache, per-tenant quotas, and cost you can actually attribute.

A GPU-heavy generative system where sampling steps dominate the bill and one bad image is a headline. The signal is the diffusion serving pipeline, the cost levers on per-image GPU time, and a mandatory two-sided safety layer.

A GPU-heavy generative-audio system with text/genre/lyric conditioning, minutes-long coherence, and copyright constraints. The signal is the generation pipeline plus the async, cost, and safety layer that makes it shippable.

Video generation is image generation plus the brutal constraint of temporal consistency, and it is the most GPU-intensive generative task there is. The signal is diffusion over spacetime plus a serving design that survives multi-minute jobs.

Batch scoring a billion items is a throughput-and-cost problem, the opposite of low-latency serving. Every choice that hurts latency helps you here. Here is how to design for dollars-per-million-documents.

GPU spend is usually the biggest line item in an AI org, and this question checks whether you think in utilization, not just capacity. The signal is squeezing per-unit cost (utilization, right-sizing, spot, quantization) before scaling out.

A Deployment and a Service will serve a model, but GPUs break every Kubernetes default: scheduling, probes, autoscaling signals, and rollouts. Here is the setup that survives production, and when KServe earns its complexity.

A hardware-literacy question, especially at NVIDIA and the labs. The signal is understanding throughput vs latency hardware, the memory hierarchy, and why matrix multiplication maps onto GPUs and TPUs. Here is the architecture-aware answer.

Autoscaling GPUs is not autoscaling web servers: GPUs are scarce and expensive, model loading is slow, and the right signal is not CPU. The signal is scaling on queue and GPU metrics, taming cold starts, and the scale-to-zero economics.

CPU-based autoscaling that works fine for a web tier quietly fails on GPU inference: wrong signal, and replicas that take minutes to warm. The interviewer wants the signals you actually scale on and how you hide the cold start.

Paying for accelerators that idle half the time is the most common waste in ML training, and the instinct to add more GPUs makes it strictly worse. The interviewer wants the profiling discipline that finds what is starving them.

The counterintuitive truth of LLM serving: token generation is limited by how fast you can read weights from memory, not by math. Once you see that, the whole optimization menu falls out of one number.

Linear scaling is the marketing number; the real curve bends early for reasons that are physics, not bugs. Here is where the speedup goes and how to claw it back.

A 256-GPU job that wedges with no error at 3am is a special kind of pain. The causes are a short, recurring list. Here is the systematic way to find which one bit you.

Spot GPUs are often 60-90% cheaper, and they vanish with two minutes' warning. The savings are only real if a preemption costs you minutes, not the run. Here is how to make that true.

Naive 4-bit quantization can tank quality, and the reflex to give up and serve fp16 leaves a big speedup on the table. The accuracy is usually recoverable. Here is the ladder of fixes.

Dedicating a GPU per model strands most of a fleet on idle silicon. Sharing safely is a real systems problem with four distinct mechanisms, each tuned to a different traffic shape. Here is how to pick.