Backpropagation, Intuitively

TL;DR: Backpropagation is just the chain rule applied in reverse. The forward pass computes the output and caches every intermediate activation; the backward pass walks the same graph from loss to inputs, multiplying local derivatives and accumulating the gradient for each parameter in a single sweep. The reason it is cheap (about the cost of two or three forward passes, not one per parameter) is that reverse-mode reuses shared sub-computations instead of recomputing them.

The chain rule, run backwards

A neural net is a long composition of functions: loss = L(f_n(...f_1(x))). To train it you need the gradient of the loss with respect to each weight. The chain rule says the derivative of a composition is the product of local derivatives along the path. The only real idea in backprop is the order you multiply those terms in.

Multiply from the inputs forward (forward-mode) and you compute, for each input, how it affects everything downstream. Multiply from the loss backward (reverse-mode) and you compute, for the single scalar loss, how it depends on everything upstream. Since a network has millions of parameters but one scalar loss, reverse-mode gives you all those gradients in roughly the cost of one backward sweep. Forward-mode would cost one sweep per parameter. That asymmetry is the whole game.

Forward pass: compute and cache

The forward pass runs the network normally and stores the activations it will need later. For a layer z = Wx + b; a = act(z), the gradient with respect to W will depend on the input x, and the gradient through the activation will depend on z (or a). So those values are kept in memory. This is why training memory is dominated by activations, not weights: a transformer's activation cache scales with batch size and sequence length, and is the reason long-context training is memory-hungry and why gradient checkpointing (recompute activations instead of storing them) exists.

Backward pass: accumulate gradients

The backward pass starts with dL/dL = 1 and propagates a gradient signal (the "upstream gradient") layer by layer toward the input. At each node it does two things: multiply the incoming gradient by the local Jacobian to pass signal further back, and combine that signal with the cached activation to produce the gradient for that node's parameters.

A worked example for one linear layer z = Wx, with upstream gradient g = dL/dz:

• gradient to the weights: dL/dW = g · xᵀ (uses the cached input x)
• gradient to pass back: dL/dx = Wᵀ · g

When a value feeds two places (a residual branch, a weight reused across timesteps in an RNN), its gradients add. "Accumulate" is literal: you sum contributions from every path the value influenced. Forgetting to zero these accumulators between steps is the classic bug behind loss.backward() followed by a missing optimizer.zero_grad().

Why it is efficient

Reverse-mode pays for this speed with memory: it must hold the forward activations until the backward pass consumes them. Time-memory tradeoff, not a free lunch.

The failure modes it exposes

Because the backward signal is a product of many Jacobians, its magnitude compounds. If the per-layer factors are mostly below 1, the product shrinks toward zero (vanishing gradients) and early layers stop learning. If they are above 1, it blows up (exploding gradients) and training diverges or returns NaNs. Backprop does not cause these; it reveals what the architecture and initialization already imply. The standard fixes (residual connections, normalization, careful init, gradient clipping, non-saturating activations) all target this Jacobian product.

Why interviewers probe this

The interviewer is screening for whether you understand training cost and memory at a mechanical level, not just "the optimizer does it." A strong answer names reverse-mode autodiff, explains the forward-cache / backward-accumulate split, and ties activation memory to the cost of long sequences or large batches. The held-back follow-up is usually "why is it memory-heavy, and what would you do at scale?" The expected move there is gradient checkpointing, plus the observation that the activation cache, not the weights, dominates.

Common misconceptions

• "Backprop is a separate learning rule." It only computes gradients; gradient descent (or Adam) uses them to update weights. They are different steps.
• "It computes the gradient one parameter at a time." It computes all parameter gradients in a single backward sweep by reusing shared sub-results.
• "The forward pass can be discarded once you have the output." You must cache activations for the backward pass, which is the bulk of training memory.
• "Vanishing gradients are a bug in backprop." Backprop computes the true gradient; the smallness comes from the architecture's repeated Jacobian product.

Key takeaways

• Backprop is reverse-mode autodiff: the chain rule multiplied from loss back to inputs.
• Forward pass caches activations; backward pass reuses them to accumulate every gradient in one sweep.
• It is cheap in time (a small constant multiple of a forward pass) but expensive in memory (activations), which motivates gradient checkpointing.
• The backward signal is a product of Jacobians, so it naturally vanishes or explodes, exposing those failure modes.