TL;DR: Cut cost by raising utilization and cutting per-unit work before adding capacity. Training: spot/preemptible instances with checkpointing, right-sized hardware, mixed precision, and efficient data loading so GPUs are never idle. Inference: quantize, use continuous batching, cache, autoscale to demand (scale to zero for spiky load), and route to the smallest model that meets quality. Track cost per training run and cost per 1k inferences, then optimize the dominant one.
How to approach it. Start from the principle: the cheapest GPU-hour is the one you do not waste, so attack utilization and per-request cost first; scaling out is the last lever, not the first. Split training cost from inference cost (different bottlenecks, different levers) and tie everything back to a per-unit metric so the optimization is measurable.
A strong answer. Measure first. Track cost per training run and cost per 1,000 inferences (or per million tokens), and find which dominates the bill. Training is bursty and capital-heavy; inference is continuous and is usually the larger long-run cost at scale. Optimize the dominant one rather than the one that is easiest to talk about.
Training cost levers.
- Spot / preemptible instances. Often 60-90% cheaper but reclaimable, so pair with checkpointing and elastic/fault-tolerant training so a preemption costs minutes, not the run. This is the biggest single training-cost lever.
- Maximize GPU utilization. Idle GPUs are pure waste. Fix data-loading bottlenecks (prefetch, more workers), use a large enough batch, and overlap compute with IO. A GPU at 35% utilization is paying full price for a third of the work.
- Mixed precision (BF16/FP16) and efficient kernels (FlashAttention) to do more per GPU-hour.
- Right-size. Do not train a bigger model or longer than the metric needs; early stopping and efficient HPO (ASHA) kill wasted runs.
Inference cost levers.
- Quantization (INT8/FP8): fewer bytes per weight, faster and cheaper per token, with quality validated against a held-out eval.
- Continuous (in-flight) batching so a new request joins the running batch the moment a sequence finishes, keeping the GPU saturated instead of waiting for a static batch to drain.
- Caching (exact and semantic) so repeated or near-duplicate requests cost nothing.
- Autoscaling to demand, including scale-to-zero for spiky/low-traffic services, plus right-sizing the instance (do not serve a small model on the biggest GPU).
- Model routing / cascades: send easy requests to a small cheap (often distilled) model and escalate only hard ones to the large model.
- Spot for batch/offline inference (with retries); reserved/committed capacity for steady baseline load.
| Workload | Best capacity type | Why |
|---|---|---|
| Steady baseline serving | Reserved / committed | Predictable load, lowest unit price |
| Variable interactive | On-demand + autoscale | Pay for actual demand |
| Training, batch inference | Spot + checkpointing | Interruptible, 60-90% cheaper |
Key takeaways.
- The first lever is utilization: a GPU at 35% is paying full price for a third of the work.
- Spot plus checkpointing is the dominant training-cost lever; reserved is for steady inference baseline only.
- Quantization, continuous batching, and caching cut per-token cost before you add a single GPU.
- Route most traffic to a small model and escalate only hard cases; one large model for everything is the most common overspend.
What interviewers probe next.
- "Spot instance risk for training?" Preemption mid-run; mitigate with frequent checkpoints and elastic training so you resume cheaply (ties to fault-tolerant training design).
- "GPU at low utilization, what do you check?" Data loading or CPU preprocessing starving the GPU, batch too small, or sync stalls; fix the pipeline before buying more GPUs.
- "Model cascade / routing?" Serve most traffic with a small/quantized/distilled model and escalate only hard cases, cutting average cost sharply.
- "Reserved vs on-demand vs spot?" Reserved for steady baseline, on-demand for variable, spot for interruptible batch and (with checkpointing) training.
Common mistakes.
- Buying more GPUs before fixing utilization (idle GPUs at full price).
- Running steady high-cost inference with no quantization, batching, or caching.
- Not autoscaling, so capacity is provisioned for peak and idle most of the time.
- Using one large model for all traffic when a small model handles most requests fine.