The Bias-Variance Tradeoff

TL;DR: Prediction error splits into bias (the model is too simple to capture the true pattern, it underfits and is wrong even on training data) and variance (the model is too sensitive to the particular training sample, it overfits and fails to generalize). Lowering one usually raises the other, so good generalization is a balance, not the extreme of either. This decomposition is the lens behind regularization, model-complexity choices, and ensembling, and the first thing to reason about when a model underperforms.

The decomposition

Imagine training the same model on many different samples of data. Bias is how far its average prediction is from the truth (a systematic error from an overly-simple model). Variance is how much its predictions swing across the different samples (sensitivity to the specific data it saw). Total error is roughly bias-squared plus variance plus irreducible noise.

• High bias (underfitting): the model is too simple, it misses the pattern and does poorly on both training and test data. (A linear model on a curvy relationship.)
• High variance (overfitting): the model is too flexible, it fits the training data, including its noise, but does poorly on new data. (A deep tree that memorizes.)

The tradeoff and how to act on it

As you increase model complexity (more parameters, less regularization, more features), bias falls but variance rises; the test error is U-shaped, with a sweet spot in the middle. The practical value is diagnosis:

• Underfitting (high bias): training error is high. Fix by adding capacity (a more complex model, more features) or training longer.
• Overfitting (high variance): training error is low but test error is much higher (a large gap). Fix with more data, regularization (L1/L2, dropout), simpler models, or ensembling (bagging averages out variance).

Look at the gap between training and validation error: a large gap signals variance/overfitting; both high signals bias/underfitting.

A modern wrinkle: double descent

The classic story says past a point more capacity always overfits. Modern over-parameterized deep nets show double descent: past the interpolation point (where the model can fit the data exactly), test error can fall again, which is why huge models generalize despite fitting the training set. The classic U still guides small/medium models; double descent extends the picture in the over-parameterized regime.

Why interviewers probe this

"Your model does badly, what do you do?" is answered through this lens, so it is foundational. A strong answer decomposes error into bias and variance, uses the train-vs-validation gap to diagnose underfitting vs overfitting, and prescribes the matching fix (capacity vs regularization/data/ensembling). It is the core ML debugging skill, and connecting it to regularization, ensembling, and double descent shows real depth.

Common misconceptions

• "A complex model is always better." More complexity lowers bias but raises variance; balance generalizes best.
• "Low training error means a good model." It can mean overfitting; check the train-vs-validation gap.
• "Underfitting and overfitting have the same fix." Underfitting needs more capacity; overfitting needs more data/regularization/simplicity.
• "Bigger always overfits." In the over-parameterized regime, double descent shows test error can fall again.

Key takeaways

• Error decomposes into bias (too simple, underfits) and variance (too sensitive, overfits); reducing one often raises the other.
• Diagnose with the train-vs-validation gap: large gap means variance, both high means bias.
• Fix underfitting with capacity; fix overfitting with data, regularization, simpler models, or ensembling.
• Double descent extends the classic U-curve for very large over-parameterized models.