Machine Learning · Insper

Random Forest

Fundamental approach

How an average of unstable trees beats every one of them

The starting point: trees are unstable

In the decision-trees lesson, trees refitted on resamples of the same e-mails started with different questions.
Similar accuracies, yet two trees disagreed on 11.8% of the e-mails.
And a vote of 50 of them scored 0.950, more than the best tree (0.919).

Breiman's idea (bagging, 1996; random forests, 2001): do not fight a tree's variance, average many different trees and let their errors cancel out.

The path of this part

Why averaging worksthe variance of an average and its floor ρσ²
Bootstrap and baggingresample, grow deep trees, vote
Decorrelate the treesdraw max_features candidates at every split
Validate out-of-bagthe out-of-bag (OOB) error; how many trees

Each stage answers one question: why does an average err less than its members? → where do many different trees come from? → how do we make the trees less alike? → how do we evaluate without a validation set?

Stage 1 · why does an average err less than its members?

Why averaging works — and where it stops

\[ \operatorname{Var}\Big(\frac{1}{n}\sum_{i=1}^n X_i\Big) = \rho\,\sigma^2 + \frac{1-\rho}{n}\,\sigma^2 \]

Variance of the mean falling with n down to the floor ρσ²; majority vote with independent and correlated voters

The second term vanishes as \(n\) grows.
The first, \(\rho\sigma^2\), does not depend on \(n\): it is the error everyone shares.
With \(\rho = 0.3\): 10 members take the variance from 1 to 0.37; 100 only to 0.31.

A forest needs reasonable members (small \(\sigma^2\)) that are weakly correlated (small \(\rho\)).

Stage 2 · where do many different trees come from?

The bootstrap: many training sets from one

Six bootstrap samples of 10 rows: how many times each row was drawn and which were left out

Draw \(n\) rows with replacement: some go in several times, others not at all.
Each row is left out with probability \((1 - 1/n)^n \to 1/e \approx 36.8\%\).

Bagging: bootstrap aggregating

Grow \(B\) trees, each on its own bootstrap sample (usually to full depth), and average their predictions (regression) or let them vote (classification).

Bagging: low bias, half the variance

Twenty refits of a shallow tree, a full tree and a bagging

	bias²	variance
tree, depth 3	0.0223	0.0274
full tree	0.0010	0.0883
bagging, 100	0.0006	0.0413

Pruning trades variance for bias. Bagging refuses the trade: it keeps the full tree's bias and halves the variance.

The vote smooths the boundary

A full tree, three trees of the forest and the vote of 300 trees on two moons

Each tree of the forest is worse than the single tree (0.858 against 0.885), yet the vote scores 0.903: each tree's islands sit in different places and lose the vote.

Simulator: why averaging trees works

Predict. Add trees from 1 to 60. Do the individual trees (left) get any better? Does the error of the vote keep falling?

What averaging does not do

It does not improve the trees: each one stays as shaky as the first; the average is what improves.
It does not remove bias: the average of shallow trees is still shallow.
It does not remove the shared variance \(\rho\sigma^2\): past a point, more trees add nothing.

n_estimators is a matter of budget, not tuning: more is never worse, only slower.

Stage 3 · how do we make the trees less alike?

Random forest: draw the candidates of every split

In bagging every root picks f3; in a random forest each root draws three candidates

At every split, only max_features drawn features are eligible. Weaker trees, but less alike.

Strength against correlation, on digits

max_features	same root	correlation	one tree's error	forest error
64 (bagging)	66%	0.304	0.195	0.053
20	24%	0.233	0.215	0.027
8 = \(\sqrt{64}\)	10%	0.198	0.249	0.023
1	4%	0.133	0.375	0.030

Worse trees, a forest more than twice as good: a smaller \(\rho\) makes up for a larger \(\sigma^2\). With 1 feature the trees get too weak. Default: \(\sqrt{d}\) for classification.

Stage 4 · how do we evaluate without a validation set?

Out-of-bag: validation for free

Each tree did not see ~37% of the rows. For each row, only the trees that did not see it vote.
Counting the errors over all rows: the OOB error, with no extra fit.

digits, 500 trees	accuracy	cost
out-of-bag	0.975	1 fit
5-fold CV	0.977 ± 0.007	5 fits
test (450 images)	0.976	—

To compare with other models, which have no OOB, use the same CV for all of them.

How many trees?

OOB, test and CV error falling until about 100 trees and then flat

The error falls, levels off and never climbs back. Only read OOB with a few dozen trees: with 5, each row gets about 2 votes and the OOB error (0.255) is far worse than the test error (0.087).

Summary

Averaging cuts variance down to the floor \(\rho\sigma^2\).
The bootstrap gives each tree a different sample; ~37% of the rows are left out.
Bagging: deep trees on bootstrap samples, averaged — low bias, variance cut.
max_features: worse, less correlated trees; a better forest.
OOB: validation for free; more trees never hurt, they only cost time.

symptom	what to do
OOB curve went flat	stop adding trees
flat, but too high	deeper trees, another `max_features`, better features

Fundamental lab · about 40 min, in pairs

A spam filter: from one tree to a forest, steps 1 to 7

Spambase: 4,601 e-mails, 57 word and character frequencies.
One tree, bagging, random forest and Extra-Trees on the same folds; OOB against CV; how many trees.
Notebook on Colab (in Portuguese): nothing to download. Save a copy to your Drive.
At every step, answer the Predict before running.
Simulations: is the forest better than its own trees? (step 4) and shallow trees, however many (step 7).

Pace: steps 1 to 3 in 10 min; steps 4 and 5 and the first simulation in 15 min; steps 6 and 7 and the second in 15 min.