numerical stability

Almost everyone can quote softmax(QKᵀ/√d_k)V. The interviewer wants the variance argument and the exact training failure the scale prevents. Here is the answer that separates memorization from understanding.

A deceptively simple ML-coding ask. Anyone can write exp/sum; the signal is the max-subtraction trick and the log-sum-exp form that keep it from overflowing. Here is the stable implementation and why the naive one breaks.

The textbook variance formula loses catastrophic precision on streaming data and cannot update incrementally. Welford's one-pass update is the standard for online feature normalization. Here it is.