Gradient Descent and Optimizers

TL;DR: Training minimizes a loss by gradient descent: compute how the loss changes with each parameter (the gradient) and take a step in the opposite direction. Using the whole dataset per step is stable but slow; using one example is fast but noisy; mini-batch SGD (a small batch) is the practical balance and exploits GPU parallelism. Momentum accelerates and smooths the path, and Adam (momentum plus per-parameter adaptive learning rates) is the default optimizer. The learning rate is the most important hyperparameter, usually scheduled with warmup then decay.

The core loop

A model has parameters and a loss measuring how wrong it is. The gradient points in the direction of steepest increase of the loss, so stepping opposite it reduces the loss. Repeat, and the parameters descend toward a good solution. How much data you use per step defines the variant:

• Batch (full) GD: use the entire dataset per step. Stable gradient, but slow and memory-bound, infeasible at scale.
• Stochastic GD (SGD): one example per step. Fast and noisy; the noise can help escape poor minima but the path is jittery.
• Mini-batch GD (the standard): a small batch (e.g. 32-512). Balances stable-enough gradients with speed, and exploits GPU parallelism, which is the real reason it dominates.

Momentum and Adam

Plain SGD can crawl through flat regions and oscillate in ravines. Momentum accumulates a velocity from past gradients, smoothing the trajectory and accelerating in consistent directions. Adam combines momentum (a running average of gradients) with per-parameter adaptive learning rates (scaling each step by recent gradient magnitude), so it works well with little tuning, which is why it (and AdamW, with proper weight decay) is the default for deep learning.

The learning rate is the key knob

The learning rate sets the step size and is usually the most impactful hyperparameter:

• Too high: the loss oscillates or diverges (NaNs).
• Too low: training is painfully slow or gets stuck.

Modern training schedules it: warmup (start small and ramp up, so early noisy gradients do not destabilize training) then decay (cosine/step, shrink the rate so the model settles into a good minimum).

Why interviewers probe this

Gradient descent underlies all training, and its failure modes are diagnosable, so "the loss is diverging" or "training stalled" are answered through this lens. A strong answer explains the gradient step, why mini-batch is the practical choice (stability plus GPU parallelism), what momentum and Adam add, and that the learning rate is the dominant knob (too high diverges, too low stalls) scheduled with warmup and decay. That connects the math to the levers you actually pull when training misbehaves.

Common misconceptions

• "Full-batch gradient descent is best." It is stable but does not scale; mini-batch balances stability, speed, and parallelism.
• "Adam is always better than SGD." Adam works well out of the box; well-tuned SGD with momentum can generalize better.
• "The learning rate is just another hyperparameter." It is usually the most important one, too high diverges, too low stalls.
• "Schedules are optional." Warmup and decay materially stabilize and improve large-model training.

Key takeaways

• Gradient descent steps parameters opposite the loss gradient to reduce error.
• Mini-batch SGD balances gradient stability, speed, and GPU parallelism, so it is the standard.
• Momentum smooths and accelerates; Adam adds per-parameter adaptive rates and is the default.
• The learning rate is the key knob (too high diverges, too low stalls), scheduled with warmup then decay.