Ensembling: Bagging, Boosting, Stacking

TL;DR: Ensembles combine several models to outperform any one, because when models make decorrelated errors, averaging or voting cancels the individual mistakes. Bagging trains models in parallel on bootstrap resamples and averages them, reducing variance (random forest). Boosting trains models sequentially, each correcting the previous ensemble's errors, reducing bias (XGBoost/LightGBM, which dominate tabular ML). Stacking trains a meta-model to combine base models' predictions. The non-negotiable requirement is diversity, ensembling identical models gains nothing.

Why combining helps

If you average models that make the same mistakes, you gain nothing. But if their errors are decorrelated (they err in different ways), the mistakes partially cancel and the combination is more accurate and stable. The math is exact for variance: average N models whose errors each have variance σ² and pairwise correlation ρ, and the ensemble variance is ρσ² + (1-ρ)σ²/N. The second term vanishes as you add models, but the first term, set by correlation, does not. That single formula is the whole game: a random forest with 500 trees does not drive variance to zero, it drives it to ρσ², so the entire job of random feature subsampling is to push ρ down. Two angles, mapped to the bias-variance decomposition: averaging reduces variance, and sequentially fitting residuals reduces bias.

The three families

• Bagging (bootstrap aggregating). Train many models in parallel on bootstrap resamples (and, in random forests, random feature subsets), then average/vote. Reduces variance, parallelizable, and hard to overfit by adding trees.
• Boosting. Train models sequentially, each fitting the errors (residuals) of the current ensemble. Reduces bias; gradient boosting (XGBoost, LightGBM) dominates tabular data, with shrinkage (learning rate), tree depth, and subsampling as the regularizers. Boosting can overfit if you add too many rounds, so you early-stop on a validation set.
• Stacking. Train diverse base models, then a meta-model on their out-of-fold predictions that learns how best to combine them, more powerful than simple averaging, the classic Kaggle winner.

Worked example: why decorrelation beats accuracy

Take 3 binary classifiers, each 70% accurate, majority vote. If their errors were independent, the vote is wrong only when 2 or 3 err: 3 * 0.3² * 0.7 + 0.3³ = 0.189 + 0.027 = 0.216, so accuracy jumps to about 78.4%, well above any single 70% model. Now make them identical (correlation 1.0): every model errs on the same examples, the vote errs exactly when they do, and accuracy stays 70%. Same component accuracy, completely different ensemble, and the only thing that changed is correlation. This is why three mediocre-but-different models often beat one carefully tuned model, and why ensembling five seeds of the same architecture barely moves the needle.

Practical notes

Ensembles cost more (train and serve multiple models, higher latency), so weigh the accuracy gain against cost; in production you often distill an ensemble into a single model for serving. For tabular problems, a well-tuned gradient-boosted ensemble is usually the strongest baseline, often beating deep learning. Diversity (model families, features, seeds) is the lever for ensemble quality, and the cheapest source of diversity is different feature subsets, not just different seeds.

Why interviewers probe this

Gradient boosting is the default for tabular ML, and the bias/variance framing of ensembling connects to model debugging, so it is broadly tested. A strong answer explains why ensembles work (decorrelated errors cancel, requiring diversity), distinguishes the three families (bagging parallel/variance, boosting sequential/bias, stacking meta-model), and notes the serving cost. The follow-up they hold in reserve is "why does adding more trees never overfit a random forest but can overfit a GBM?", which is the variance-vs-bias distinction made concrete.

Common misconceptions

• "Ensembling always helps." Only if the models are diverse (decorrelated errors); identical models gain nothing, as the vote example shows.
• "Bagging and boosting are the same." Bagging is parallel and reduces variance; boosting is sequential and reduces bias and can overfit.
• "More trees can overfit a random forest." Adding bagged trees only reduces variance toward ρσ²; it is boosting rounds that overfit.
• "Stacking is just averaging." Stacking trains a meta-model to learn the combination, using out-of-fold predictions to avoid leakage.
• "Deep learning beats ensembles on tabular data." Gradient-boosted trees usually win on tabular problems.

Key takeaways

• Ensembles beat single models when errors are decorrelated; ensemble variance floors at ρσ², so diversity is the lever.
• Bagging averages parallel bootstrap models (reduces variance, hard to overfit); boosting fits errors sequentially (reduces bias, early-stop).
• Stacking trains a meta-model on out-of-fold predictions to combine base models.
• Gradient boosting dominates tabular ML; weigh ensemble serving cost or distill to one model.