Machine Learning · Insper
Random Forest
Advanced approach
Where averaging stops helping, which knobs matter, and how (not) to read a forest
The path of this part
- Why the vote worksCondorcet's jury; the floor ρσ², measured
- Tune what matters
max_features, min_samples_leaf, enough trees
- Read the importancesimpurity bias, permutation, shared credit
- Variants and limitsExtra-Trees, no extrapolation, cost
Each stage answers one question: when does a vote of many beat one? → which knobs are worth tuning? → which features does the forest use, and can we trust the answer? → what can a forest not do, and what does it cost?
Stage 1 · when does a vote of many beat one?
Condorcet's jury (1785)
Each juror is right with \(p = 0.6\), independently. With three: \(0.6^3 + 3 \times 0.6^2 \times 0.4 = 0.648\).
| voters | 1 | 11 | 101 | 1,001 |
|---|
| independent (\(c = 0\)) | 0.601 | 0.755 | 0.980 | 1.000 |
| copy with prob. \(c = 0.1\) | 0.601 | 0.743 | 0.917 | 0.998 |
| copy with prob. \(c = 0.3\) | 0.601 | 0.678 | 0.618 | 0.597 |
With \(c = 0.3\), two voters correlate at only 0.09 — and a committee of 1,001 is right as often as a single voter: the copying block decides every vote.
Why bagging only halved the variance
- One bagged tree: variance \(\sigma^2 = 0.0937\).
- Two of them, at the same point: correlation \(\rho = 0.44\) — they all saw the same 100 points, noise included.
\[ \rho\sigma^2 + \frac{(1-\rho)\,\sigma^2}{100} = 0.44 \times 0.0937 + \frac{0.56 \times 0.0937}{100} = 0.0417 \]
Measured: 0.0413.
Almost all the remaining variance is the floor \(\rho\sigma^2\). More trees do not help; less correlated trees do — the job of max_features.
Stage 2 · which knobs are worth tuning?
Hyperparameters: one at a time, on digits
| knob | values (CV accuracy) | range |
|---|
n_estimators | 10: 0.941 · 100: 0.976 · 300: 0.977 | 0.036 |
max_features | 1: 0.970 · sqrt: 0.977 · None: 0.947 | 0.030 |
min_samples_leaf | 1: 0.977 · 10: 0.947 · 30: 0.915 | 0.062 |
max_depth | 3: 0.888 · 10: 0.977 · None: 0.977 | 0.089 |
criterion | gini: 0.977 · entropy: 0.979 | 0.002 |
Forests want deep trees. Tune max_features and min_samples_leaf; pick "enough trees" from the OOB curve; the rest barely matters.
Stage 3 · which features does the forest use, and can we trust the answer?
Impurity importance: biased
German credit + two columns of pure noise:
| feature | distinct values | MDI rank | permutation rank |
|---|
| credit_amount | 921 | 1 | 8 |
| random_id (noise) | 1,000 | 2 | 14 |
| checking_status | 4 | 5 | 1 |
| coin (noise) | 2 | 17 | 22 |
MDI is measured on the training data and rewards columns with many thresholds, which fit noise deep in the trees. Prefer permutation on held-out data.
Correlated features share the credit
| MDI | permutation |
|---|
| checking_status, alone | 0.086 | 0.144 |
| checking_status, with a twin | 0.057 | 0.049 |
| checking_copy | 0.059 | 0.054 |
Shuffling one copy lets the other do the work. A feature that looks unimportant may just have a substitute: permute the whole group at once.
Stage 4 · what can a forest not do, and what does it cost?
Extra-Trees, and what a forest cannot do
Extra-Trees
Random thresholds instead of searched ones; no bootstrap. On digits: 0.981 against the forest's 0.977 (a tie), with twice the nodes per tree.
No extrapolation
Trained on \(y = 3x\), \(0 \le x \le 10\): it predicts 29.6 at \(x = 12\) and at \(x = 20\). The linear model: 59.7 (truth: 60).
Every prediction is an average of training targets: outside the range seen, the forest goes flat.
The cost of a forest
| digits | accuracy | nodes | saved size |
|---|
| one tree | 0.829 | 293 | 0.04 MB |
| forest, 300 trees | 0.977 | 98,696 | 14.32 MB |
| forest, leaves ≥ 3 | 0.971 | 65,754 | 9.57 MB |
Bagging builds trees independently, in parallel, and cuts variance. Boosting (next lesson) builds them in sequence and cuts bias.
Advanced lab · about 30 min, in pairs
Tune, pick the threshold and test, steps 8 to 11
- Same notebook, same session: the steps use
X_train, y_train, cv, folds and roc_auc_score.
- Tune
max_features and min_samples_leaf; a threshold that sends at most 1% of legitimate e-mail to spam, chosen out-of-bag; the test, once; where the forest looks.
- Notebook on Colab (in Portuguese), from step 8.
- Simulations: the search for free, with OOB (step 8) and the importance of a pair of twins (step 11).
Pace: step 8 and the first simulation in 10 min; steps 9 and 10 in 10 min; step 11 and the second in 10 min.