TL;DR: Mixed precision stores and computes most tensors in 16-bit (FP16 or BF16) for speed and memory, while keeping a master copy of weights and optimizer states in FP32 for stable updates. FP16 has little dynamic range, so small gradients underflow to zero; loss scaling multiplies the loss (and thus gradients) up before backward and unscales before the update. BF16 has FP32-like range (fewer mantissa bits), so it usually needs no loss scaling, which is why it is preferred on supporting hardware.
How to approach it. Anchor on the distinction between dynamic range (how large or small a magnitude the format can represent) and precision (how finely it resolves nearby values). FP16 and BF16 differ exactly there, and that difference explains loss scaling and why BF16 won. Then cover what stays FP32 and why.
A strong answer. Why 16-bit. Half precision halves memory for weights, activations, and gradients and runs faster on tensor-core hardware, so you fit bigger models and batches and train faster. But naive all-FP16 training diverges, so you mix precisions.
FP16 vs BF16. Both are 16-bit but split the bits differently:
| Format | Exponent | Mantissa | Range | Precision |
|---|---|---|---|---|
| FP32 | 8 | 23 | wide | high |
| FP16 | 5 | 10 | narrow | medium |
| BF16 | 8 | 7 | wide (FP32-like) | coarse |
FP16 has decent precision but a narrow dynamic range, so very small values (many gradients) underflow to zero and very large ones overflow. BF16 keeps FP32's 8 exponent bits but only 7 mantissa bits: FP32-like range, coarser precision. For deep learning, range matters more than fine resolution, so BF16 trains stably without the tricks FP16 needs, which is why it became the default where hardware supports it.
Loss scaling (FP16's fix). Because FP16 gradients underflow, you scale the loss up by a large factor before backprop. By the chain rule every gradient is multiplied by that factor, pushing small ones into FP16's representable range. You then unscale the gradients before the optimizer step so the update is correct. Dynamic loss scaling auto-tunes the factor: raise it while stable, and on an overflow (inf/nan) skip that step and lower it. BF16's wide range means small gradients do not underflow, so loss scaling is usually unnecessary.
What stays FP32 (the "mixed"). Keep an FP32 master copy of the weights and the optimizer states (Adam's momentum and variance). Updates are tiny relative to weights, and accumulating them in FP16 loses them to rounding (the update vanishes), so you apply updates to the FP32 master and cast to 16-bit for the forward and backward passes. Reductions and accumulations (softmax sums, batch-norm statistics, the loss) are also often done in FP32 for stability.
The defensible framing: 16-bit for compute and memory, FP32 master weights and optimizer states for stable updates; BF16 if available (its range solves the underflow), FP16 with dynamic loss scaling otherwise.
Key takeaways
- The whole story is range vs precision: FP16 trades range for mantissa bits, BF16 keeps FP32 range and gives up precision.
- FP16 needs loss scaling because small gradients underflow; BF16's wide exponent usually makes scaling unnecessary.
- FP32 master weights and optimizer states exist so tiny updates are not rounded away.
- NaN means overflow, a stalled loss often means underflow; dynamic loss scaling and BF16 are the fixes.
What interviewers probe next.
- "Why does FP16 need loss scaling but BF16 usually does not?" BF16 has FP32-range exponent bits, so small gradients stay representable; FP16's narrow range underflows them without scaling.
- "Why keep FP32 master weights?" Weight updates are small; accumulating them in 16-bit rounds them away, so the model stops learning. The master copy preserves them.
- "What is FP8 and where does it fit?" Even smaller (training and inference on the newest hardware); needs careful per-tensor scaling and is an active frontier.
- "Symptom of bad mixed precision?" Loss goes to NaN/inf (overflow) or stops improving (underflow); dynamic loss scaling and BF16 address these.
Common mistakes.
- Saying FP16 and BF16 are interchangeable; they trade range vs precision, which is the whole point.
- Forgetting loss scaling for FP16, then watching gradients underflow to zero.
- Not keeping FP32 master weights and optimizer states, so updates vanish to rounding.
- Treating mixed precision as "just cast everything to 16-bit," which diverges.