Machine Learning · Insper

Random Forest

Advanced approach

Where averaging stops helping, which knobs matter, and how (not) to read a forest

The path of this part

Why the vote worksCondorcet's jury; the floor ρσ², measured
Tune what mattersmax_features, min_samples_leaf, enough trees
Read the importancesimpurity bias, permutation, shared credit
Variants and limitsExtra-Trees, no extrapolation, cost

Each stage answers one question: when does a vote of many beat one? → which knobs are worth tuning? → which features does the forest use, and can we trust the answer? → what can a forest not do, and what does it cost?

Stage 1 · when does a vote of many beat one?

Condorcet's jury (1785)

Each juror is right with \(p = 0.6\), independently. With three: \(0.6^3 + 3 \times 0.6^2 \times 0.4 = 0.648\).

voters	1	11	101	1,001
independent (\(c = 0\))	0.601	0.755	0.980	1.000
copy with prob. \(c = 0.1\)	0.601	0.743	0.917	0.998
copy with prob. \(c = 0.3\)	0.601	0.678	0.618	0.597

With \(c = 0.3\), two voters correlate at only 0.09 — and a committee of 1,001 is right as often as a single voter: the copying block decides every vote.

Simulator: a jury that copies

Predict. With \(c = 0.3\), is a jury of 1,001 right more or less often than one voter alone? And with \(c = 0.1\)?

Why bagging only halved the variance

One bagged tree: variance \(\sigma^2 = 0.0937\).
Two of them, at the same point: correlation \(\rho = 0.44\) — they all saw the same 100 points, noise included.

\[ \rho\sigma^2 + \frac{(1-\rho)\,\sigma^2}{100} = 0.44 \times 0.0937 + \frac{0.56 \times 0.0937}{100} = 0.0417 \]

Measured: 0.0413.

Almost all the remaining variance is the floor \(\rho\sigma^2\). More trees do not help; less correlated trees do — the job of max_features.

Stage 2 · which knobs are worth tuning?

Hyperparameters: one at a time, on digits

knob	values (cross-validation, CV, accuracy)	range
`n_estimators`	10: 0.941 · 100: 0.976 · 300: 0.977	0.036
`max_features`	1: 0.970 · sqrt: 0.977 · None: 0.947	0.030
`min_samples_leaf`	1: 0.977 · 10: 0.947 · 30: 0.915	0.062
`max_depth`	3: 0.888 · 10: 0.977 · None: 0.977	0.089
`criterion`	gini: 0.977 · entropy: 0.979	0.002

Forests want deep trees. Tune max_features and min_samples_leaf; pick "enough trees" from the out-of-bag (OOB) curve; the rest barely matters.

Simulator: which knobs matter?

Predict. Which knob has the largest range? And does that mean it is worth moving?

Stage 3 · which features does the forest use, and can we trust the answer?

Impurity importance: biased

German credit + two columns of pure noise. MDI (mean decrease in impurity): the impurity reduction each feature adds up across the trees, measured on training data.

feature	distinct values	MDI rank	permutation rank
credit_amount	921	1	8
random_id (noise)	1,000	2	14
checking_status	4	5	1
coin (noise)	2	17	22

MDI is measured on the training data and rewards columns with many thresholds, which fit noise deep in the trees. Prefer permutation on held-out data.

Correlated features share the credit

	MDI	permutation
checking_status, alone	0.086	0.144
checking_status, with a twin	0.057	0.049
checking_copy	0.059	0.054

Shuffling one copy lets the other do the work. A feature that looks unimportant may just have a substitute: permute the whole group at once.

Simulator: MDI against permutation

Predict. Where does the serial number (noise) rank under each measure? What happens to checking_status when it gets a copy?

Stage 4 · what can a forest not do, and what does it cost?

Extra-Trees, and what a forest cannot do

Extra-Trees

Random thresholds instead of searched ones; no bootstrap. On digits: 0.981 against the forest's 0.977 (a tie), with twice the nodes per tree.

No extrapolation

Trained on \(y = 3x\), \(0 \le x \le 10\): it predicts 29.6 at \(x = 12\) and at \(x = 20\). The linear model: 59.7 (truth: 60).

Every prediction is an average of training targets: outside the range seen, the forest goes flat.

Simulator: Extra-Trees on the 12 churn customers

No bootstrap; feature and threshold drawn. Predict. Will a random threshold land near the best split, bill ≤ 82.5?

The cost of a forest

digits	accuracy	nodes	saved size
one tree	0.829	293	0.04 MB
forest, 300 trees	0.977	98,696	14.32 MB
forest, leaves ≥ 3	0.971	65,754	9.57 MB

Bagging builds trees independently, in parallel, and cuts variance. Boosting (next lesson) builds them in sequence and cuts bias.

Advanced lab · about 30 min, in pairs

Tune, pick the threshold and test, steps 8 to 11

Same notebook, same session: the steps use X_train, y_train, cv, folds and roc_auc_score.
Tune max_features and min_samples_leaf; a threshold that sends at most 1% of legitimate e-mail to spam, chosen out-of-bag; the test, once; where the forest looks.
Notebook on Colab (in Portuguese), from step 8.
Simulations: the search for free, with OOB (step 8) and the importance of a pair of twins (step 11).

Pace: step 8 and the first simulation in 10 min; steps 9 and 10 in 10 min; step 11 and the second in 10 min.