Compare SGD, momentum, RMSProp, Adam, and AdamW. Why does AdamW decouple weight decay?

TL;DR: SGD steps down the gradient; momentum adds a velocity term to accelerate along consistent directions and damp oscillation; RMSProp scales each parameter's step by a running average of its squared gradients (per-parameter adaptive rates); Adam combines momentum and RMSProp. AdamW fixes a real defect: in Adam, L2 regularization got entangled with the adaptive scaling so it did not act as true weight decay; AdamW applies weight decay directly to the weights, decoupled from the gradient step, which generalizes better and is now standard.

How to approach it. Walk the lineage as a progression (each adds something: momentum, then per-parameter scaling, then both), then explain the AdamW fix precisely, because that decoupling is the part interviewers actually probe and most people get hand-wavy about. If you can state where the λθ term goes in plain Adam, you have answered the real question.

A strong answer. The progression builds one idea at a time:

• SGD. Update θ ← θ − lr · g. Simple, well understood, often generalizes best with tuning, but slow and sensitive to learning rate, and it struggles in ravines (oscillates across steep directions, crawls along shallow ones).
• Momentum. Maintain a velocity v ← βv + g, step with v. It accelerates along directions of consistent gradient and damps oscillation across steep directions, like a heavy ball rolling downhill. Faster and steadier than plain SGD.
• RMSProp. Keep a running average of squared gradients per parameter and divide the step by its square root, giving each parameter its own effective learning rate. This handles features and parameters with very different gradient scales (helpful for sparse or ill-conditioned problems).
• Adam. Combine the two: a momentum term (first moment) and an RMSProp-style per-parameter scaling (second moment), with bias correction for the early steps. A strong default that works out of the box, which is why it dominates, though SGD with momentum sometimes generalizes better on vision tasks with careful tuning.

The AdamW fix (the crux). The standard way to regularize is weight decay, which mathematically equals adding an L2 penalty λ‖θ‖² to the loss, so its gradient λθ is added to g. In plain Adam, that λθ term goes through the adaptive per-parameter scaling (it gets divided by the running gradient magnitude like everything else), so parameters with large gradients effectively get less weight decay. That couples regularization strength to gradient history, which is not what weight decay is supposed to do, and it hurts generalization. AdamW decouples them: apply the Adam step from the gradient, then separately shrink the weights by lr · λ · θ directly, so weight decay is uniform and independent of the adaptive scaling. This consistently generalizes better and is now the default for transformers.

Key takeaways

• The lineage is additive: momentum (velocity), then RMSProp (per-parameter 1/√v scaling), then Adam (both, plus bias correction).
• Adam's adaptivity is per-parameter learning rates from the running squared-gradient average; that is the line most candidates miss.
• In plain Adam the λθ penalty passes through the adaptive scaling, so decay is uneven; AdamW applies it straight to the weights.
• Adam is the safe default; SGD+momentum can still win on vision with tuning because its noise finds flatter minima.

What interviewers probe next.

• "Why does Adam sometimes generalize worse than SGD?" The adaptive rates can converge to sharper minima; SGD's noise can find flatter, better-generalizing ones. Hence SGD+momentum still wins some benchmarks with tuning.
• "What does bias correction do in Adam?" The moment estimates start at zero and are biased low early; the correction rescales them so early steps are not too small.
• "In plain Adam, why isn't L2 true weight decay?" Because the L2 gradient term passes through the per-parameter 1/√v scaling, so decay strength varies by gradient magnitude; AdamW applies it directly to weights instead.
• "How do learning-rate schedules interact?" Warmup (ramp up to avoid early instability) then decay (cosine) is standard with AdamW for transformers.

Common mistakes.

• Describing Adam as "just a better SGD" without naming momentum + per-parameter scaling.
• Being unable to explain the AdamW decoupling precisely (the L2-through-adaptive-scaling problem).
• Claiming Adam always beats SGD; SGD+momentum can generalize better with tuning.
• Forgetting bias correction or the role of the squared-gradient (second moment) term.