Decision Trees and Splitting Criteria

TL;DR: A decision tree greedily splits the data on the feature and threshold that most reduce node impurity (Gini or entropy), recursing until a stopping rule fires. A single tree is fully interpretable (you can trace the exact path to a prediction) but high-variance: small data changes redraw the splits. That instability is exactly why trees shine as ensemble members, where averaging many trees cancels the variance.

How a tree decides where to split

Training is greedy and local. At each node, the algorithm scans every feature and every candidate threshold, scores how much each split would purify the resulting children, and keeps the best one. Then it recurses on each child. "Purify" means making each child node closer to a single class (classification) or low-variance (regression).

The two standard impurity measures for classification:

• Gini impurity: 1 - Σ pₖ², the probability of misclassifying a random sample if you labeled it by the node's class distribution. Zero when the node is pure.
• Entropy: -Σ pₖ·log₂(pₖ), the information-theoretic uncertainty. The split's information gain is the entropy drop from parent to weighted children.

Worked example. A node has 8 positives and 8 negatives. Its Gini is 1 - (0.5² + 0.5²) = 0.5. Suppose a split sends {6 pos, 2 neg} left and {2 pos, 6 neg} right. Each child's Gini is 1 - (0.75² + 0.25²) = 0.375. The weighted child impurity is 0.5·0.375 + 0.5·0.375 = 0.375, an improvement of 0.5 - 0.375 = 0.125. The split that maximizes that drop wins. Gini and entropy almost always pick the same splits; Gini is the default in most libraries because it skips the logarithm.

Depth, pruning, and the overfitting knob

Left unconstrained, a tree will keep splitting until every leaf is pure, memorizing the training set including its noise. Depth is the central bias-variance dial:

• Shallow tree: high bias, low variance, may underfit.
• Deep tree: low bias, high variance, almost always overfits.

Two ways to control it. Pre-pruning (early stopping) caps growth with max_depth, min_samples_leaf, or a minimum impurity decrease. Post-pruning grows the full tree, then collapses splits that do not pay their way on a validation set (cost-complexity pruning trades leaf count against accuracy via a penalty α). Post-pruning tends to generalize better because the tree can find a good split beneath a mediocre one before deciding to keep the branch.

Why a single tree is interpretable but fragile

The interpretability is real: a prediction is just a path of yes/no questions, so you can show a stakeholder exactly why a loan was flagged. Each leaf is a human-readable rule.

The fragility is also real. Because splits are chosen greedily on thresholds, changing a few training rows can flip the top split and cascade into a completely different tree. That high variance means a lone tree rarely tops a leaderboard.

The bridge to ensembles

The fix follows directly from the failure mode. If one tree is unbiased-ish but high-variance, average many decorrelated trees and the variance shrinks while the bias stays put. That is the entire idea behind ensembling:

• Random forests train many deep trees on bootstrapped samples and random feature subsets, then average. The randomness decorrelates the trees so averaging actually helps.
• Gradient boosting (XGBoost, LightGBM, CatBoost) goes the other way: many shallow trees fit sequentially, each correcting the previous ensemble's residuals. These dominate tabular ML competitions.

Single trees still earn their keep when you need a literal decision flowchart a non-technical reviewer can audit, or as a fast, debuggable baseline before reaching for a forest.

Why interviewers probe this

Trees test whether you connect a splitting criterion to the bias-variance story. The strong-answer move is to compute an impurity drop on the spot, then explain that a single tree's variance is the reason ensembles exist, not an unrelated fact. The held-back follow-up is often "why does a random forest use random feature subsets?" (to decorrelate trees so averaging reduces variance) or "boosting versus bagging, which trees are deep and why?" Candidates who only recite "Gini measures impurity" without linking depth to overfitting get screened out here.

Common misconceptions

• "Gini and entropy give very different trees." They almost always agree on splits; the difference is negligible in practice, and Gini is faster.
• "Deeper trees are more accurate." Only on training data. Past a point, depth adds variance and hurts test accuracy.
• "Decision trees need feature scaling." They do not. Splits are threshold-based, so monotonic transforms (including scaling) leave the tree unchanged.
• "Random forests are just bigger decision trees." They are an average of many trees, with bootstrapping and feature randomness designed specifically to decorrelate them.

Key takeaways

• A tree splits greedily on whichever feature/threshold most reduces Gini or entropy, recursing until a stopping rule fires.
• Depth is the bias-variance knob; control overfitting with pre-pruning (max_depth, min_samples_leaf) or cost-complexity post-pruning.
• A single tree is interpretable (trace the path) but high-variance, which is precisely why it is the unit of random forests and gradient boosting.
• Trees ignore feature scaling and handle mixed feature types, but a lone tree rarely beats an ensemble on predictive accuracy.