Machine Learning · Insper

Decision Trees

Advanced approach

Where trees shine, where they fail, and why they became the raw material of ensembles

The path of this part

Measure the splitimpurity, not errors; variance for regression
Prune after growingcost-complexity pruning and the choice of α
Face the instabilitya small change in the data, a different tree
Strengths and limitsscaling, staircases, NaN, importance

Each stage answers one question: what should a split measure? → how much of a grown tree should we keep? → how far can one tree be trusted? → when is a tree the right tool?

Stage 1 · what should a split measure?

Why not simply count errors?

Error rate

share outside the node's majority class

\[ 0.20 \;\to\; \tfrac{5}{10}\cdot 0 + \tfrac{5}{10}\cdot 0.40 = 0.20 \]

does not move: the split looks useless

Gini

$1 - \sum_c p_c^2$, $p_c$ = share of class $c$ in the node

\[ 0.32 \;\to\; \tfrac{5}{10}\cdot 0 + \tfrac{5}{10}\cdot 0.48 = 0.24 \]

drops: one child is already solved

Why Gini sees it and the error does not

$p$ = share of positives in the node. ■ the parent on the curve ($p = 0.2$); ● the children ($p = 0$ and $0.4$); ○ the children's weighted average, in the middle of the dashed chord.
The error rate is a straight line up to 0.5: the chord lies on it and the drop is zero. Gini is curved (strictly concave): the chord passes below and the drop shows.

A concave criterion rewards progress before the prediction changes. That is what makes the greedy search work.

Regression trees

A leaf predicts the mean of the target; the impurity is the mean squared error (MSE) around it, the variance of the target in the node: $ I = \frac{1}{n}\sum_{i} (y_i - \bar y)^2 $, with $n$ flats in the node and $\bar y$ their mean.

Eight flats, rent against area. Before: a single mean, R$ 2.362 thousand, long residuals, MSE 0.787. After the split area ≤ 67.5: mean 1.720 for the 5 small ones and 3.433 for the 3 large ones, short residuals, weighted MSE 0.099

The split is scored as before, by the children's weighted average: 0.787 → 0.099. One question removes 87% of the variance (0.688 / 0.787). Same algorithm, another impurity.

A staircase that cannot extrapolate

Top, the depth-3 tree with 8 numbered leaves, from R$ 1.18 to R$ 3.87 thousand. Bottom, its prediction against area: 8 steps, one per leaf, in the same color; above 116 m² everything falls in leaf 8 and the prediction stays flat at 3.87 while new rents rise with the line

Each leaf is a step. Above 116 m², everything falls in leaf 8: outside the range, the RMSE (root mean squared error) is 1.14, against 0.22 for the line.

Stage 2 · how much of a grown tree should we keep?

Post-pruning: the weakest link, step by step

Grow the whole tree, then charge $\alpha$ per leaf: $R_\alpha(T) = R(T) + \alpha|T|$, with $R(T)$ the leaves' Gini weighted by their share of rows. Collapsing $t$ pays off once $\alpha \ge \alpha_{\text{eff}}(t) = \frac{R(t) - R(T_t)}{|T_t| - 1}$.

Pruning on Spambase

Training and validation accuracy by depth and by number of leaves when pruning

Unpruned: 221 leaves, 0.906. Best $\alpha$: 109 leaves, 0.915.
One-standard-error rule (Breiman et al., 1984): 57 leaves, 0.913 — a quarter of the tree.

Pruning buys little accuracy here; its real win is size: just as accurate and far easier to read.

Stage 3 · how far can one tree be trusted?

Instability: a small change, a different tree

Three trees from bootstrap resamples with different boundaries

50 bootstrap resamples of Spambase:

the root changes: $ in 29, ! in 19, two other words once each;
136 to 186 leaves;
accuracy 0.895 on average, yet two trees disagree on 11.8% of the e-mails.

Letting the 50 vote gives 0.950, better than the best single tree (0.919). Instability is a tree's flaw and the ensemble's raw material.

Stage 4 · when is a tree the right tool?

What trees do not need

Scaling

A split only uses the order. Standardizing: the same predictions on 100% of the test set. log1p: 99.3% (only the thresholds between values move).

Interactions

"Tenure matters when the bill is high": the tree says it by nesting questions, without a product feature.

Imputation

Since scikit-learn 1.3, NaN goes straight in: each split learns which way to send missing values.

Where they struggle: diagonals become staircases

A diagonal boundary approximated by steps; logistic regression and a tree given the x2 − x1 feature

16 leaves and 0.932 for what the logistic regression says with one line (0.985). With the feature $x_2 - x_1$, 2 leaves and 1.000: feature engineering helps a tree more than any hyperparameter.

Feature importance, and its bias

feature_importances_ adds up the impurity decrease of each feature's splits, on the training data.
A column of random numbers ranks 12th of 59, ahead of 46 real features; a random coin ranks 41st.
Features with many values offer many thresholds; one of them separates a few noisy rows by chance (Strobl et al., 2007).

To measure importance for real: permutation on held-out data (next lesson) or SHAP (Explainability).

Practical profile

Strengths	readable rules; no scaling; automatic interactions; accepts `NaN`; fast
Weaknesses	overfits without pruning; unstable; staircases for diagonals; cannot extrapolate; biased importance; coarse probabilities
Use when	the model must be read, audited or applied by hand

One tree, rarely; many trees, all the time: random forests (many deep trees, averaged) and gradient boosting (many shallow ones, added up).

Advanced lab · about 35 min, in pairs

The final key, and what it is worth, steps 6 to 10

Same notebook, same session: the steps use X, X_train, y_train, cv and cv_report.
Tune depth, leaves and pruning; compare with logistic regression and k-NN on the same folds; test once; measure how stable the key is.
Notebook on Colab (in Portuguese), from step 6.
Simulations: give the tree the diagonal (step 7) and bagging by hand (step 10).

Pace: steps 6 and 7 and the first simulation in 15 min; steps 8 and 9 in 10 min; step 10 and the second in 10 min.