TL;DR: Expected test error decomposes into bias² + variance + irreducible noise. Bias is error from too-simple assumptions (underfitting); variance is sensitivity to the particular training set (overfitting). Diagnose from the gap between train and validation error: high train error means high bias, a large train-to-val gap means high variance. Fix each with opposite moves.
How to approach it. Give the decomposition, then immediately make it operational: how you read train vs validation error to tell which problem you have, and the specific levers for each. Interviewers are checking whether you can debug a real model, not recite a textbook line.
A strong answer. For a model's prediction, expected test error decomposes as error = bias² + variance + irreducible noise. Bias is the error from approximating a complex reality with a simpler model (a linear fit to a curved relationship); high bias means the model underfits and misses signal. Variance is how much the learned model would change if you trained it on a different sample; high variance means it fits noise and overfits. Adding capacity lowers bias but raises variance; the sweet spot matches capacity to the data so total error bottoms out.
The practical core is diagnosis from the error curve. Read the train error and the train-to-val gap together:
| Symptom | Diagnosis | Fix |
|---|---|---|
| High train error, val close to it | High bias / underfit | More capacity, richer features, less regularization, train longer |
| Low train error, large gap to val | High variance / overfit | More data, stronger regularization (L2/L1, dropout), simpler model, early stopping |
| Both high and close, near a floor | Near irreducible noise | Stop tuning; chasing it is wasted effort |
One nuance worth raising: modern deep nets exhibit double descent, where past the interpolation threshold test error can fall again. The classic U-curve is the right mental model for most tabular and classical settings, but it is not the whole story for heavily overparameterized networks.
Key takeaways
- Diagnose from two numbers: train error tells you bias, the train-to-val gap tells you variance.
- The fixes are opposites; applying the wrong one (regularizing an underfit model) makes things worse.
- More data attacks variance, not bias. A model too simple stays wrong no matter how much data you feed it.
- The U-curve holds for classical models; overparameterized nets can show double descent.
What interviewers probe next.
- "More data: helps which one?" Primarily variance. It does little for bias; a model too simple stays wrong with more data.
- "How does regularization shift the tradeoff?" It raises bias slightly to cut variance, trading a bit of fit for better generalization.
- "Bagging vs boosting through this lens?" Bagging (random forests) mainly reduces variance by averaging; boosting mainly reduces bias by sequentially correcting errors (and can overfit if unchecked).
- "How do you measure it concretely?" Learning curves (train and val error vs training-set size) make bias vs variance visually obvious.
Common mistakes.
- Stopping at the definition with no diagnosis-from-error-curves plan.
- Prescribing "more data" for a high-bias model, which barely moves.
- Confusing the two fixes (adding regularization to an underfit model makes it worse).
- Treating the U-curve as universal and ignoring that overparameterized nets behave differently.