Machine Learning · Insper

Gradient Boosting

Fundamental approach

Weak trees, one after another, each fixing what the previous ones got wrong

The path of this part

Add, do not averagesmall trees in sequence remove bias
Boost by handfit the residuals, add them shrunk
Residuals as a gradientgradient descent on the predictions
Brake and stoplearning rate and early stopping

Each stage answers one question: why add trees instead of averaging them? → what does each new tree learn? → why is it called gradient boosting? → how many trees, and how big a step?

Stage 1 · why add trees instead of averaging them?

Average or add?

A stump, a forest of stumps, a boosting of stumps and a forest of deep trees on a sine

200 points, a sine	test MSE
one stump	0.608
forest of 200 stumps	0.572
boosting of 200 stumps	0.280
one deep tree	0.509
forest of 200 deep trees	0.369

Noise floor: 0.25.

Averaging removes variance, not bias: 200 stumps averaged are still one step. Adding removes bias: stumps fitted to the errors follow the curve.

Two strategies, same building block

Random forest in parallel and gradient boosting in sequence

Stage 2 · what does each new tree learn?

Boosting by hand: six points

\(x\)	1	2	3	4	5	6
\(y\)	2	3	4	9	10	14

Start with a constant: the mean, \(F_0 = 7\).
Compute the residuals \(y - F_0\): −5, −4, −3, 2, 3, 7.
Fit a stump to the residuals, not to \(y\): the best split is \(x < 3.5\), leaves −4 / +4 (SSE 16; the runner-up, 37).
Add it, shrunk by \(\eta = 0.5\): \(F_1 = 5, 5, 5, 9, 9, 9\). Repeat.

The boosting loop

The loop: predictions, pseudo-residuals, fitted stump, shrunk sum, with the numbers of round one

Three rounds

F0 to F3 with training MSE 18.67, 6.67, 2.92 and 1.23 and the stumps fitted to the residuals

Training MSE: 18.67 → 6.67 → 2.92 → 1.23. Round 2 isolates the last point (residual 5); round 3 repeats the split of round 1: nothing forbids it. GradientBoostingRegressor gives the same splits and leaves.

Simulator: watch the residual shrink

Predict. The same loop on 60 points. Set ν = 1 with 6 stumps, then ν = 0.15 with 40: which one fits better, and how does each get there?

Stage 3 · why is it called gradient boosting?

Why "gradient": residuals are a gradient

\[ L(y, F) = \tfrac{1}{2}(y - F)^2, \qquad -\frac{\partial L}{\partial F} = y - F \]

The residual is the negative gradient of the loss with respect to the prediction.
Boosting is gradient descent on the predictions; the tree turns the gradient, known only at the training points, into a function of \(x\).


F0(x) = the constant that minimizes Σ L(yᵢ, c)
for m = 1 .. M:
    rᵢ = −∂L(yᵢ, F)/∂F at F = F_{m−1}(xᵢ)     # pseudo-residuals
    fit a small tree h_m to (xᵢ, rᵢ)
    F_m(x) = F_{m−1}(x) + η · h_m(x)

Stage 4 · how many trees, and how big a step?

Learning rate and number of trees

Validation RMSE against the number of trees for four learning rates and four tree sizes

\(\eta = 1\): lowest after 16 trees (54.8 thousand), then climbs to 60.7 while training falls to 0.9: overfitting by adding trees.
A smaller \(\eta\) goes further: 46.9 with \(\eta = 0.3\), 45.5 with \(\eta = 0.1\) (714 trees).

Early stopping picks the number of trees

It holds out a slice of the training data (validation_fraction), measures the loss after each tree, and stops after n_iter_no_change rounds without improvement.

California housing	trees kept	validation RMSE
\(\eta = 1.0\)	41	56.4
\(\eta = 0.3\)	87	47.4
\(\eta = 0.1\)	337	46.0

The number of trees becomes an output. With a large \(\eta\) the extra trees after the minimum already overfit; give small rates more patience.

Summary

Averaging cuts variance; adding cuts bias.
Each tree is fitted to the residuals and enters shrunk: \(F_m = F_{m-1} + \eta\, h_m\).
The residuals are the negative gradient of the loss.
More trees can hurt; a smaller \(\eta\) needs more trees and goes further.
Early stopping picks the number of trees.

symptom	what to do
validation lowest early, then rising	lower \(\eta\), switch on early stopping
metrics too good to be true	look for a column that knows the future

Fundamental lab · about 35 min, in pairs

Who will subscribe? Steps 1 to 5

A Portuguese bank's telemarketing: 45,211 clients, 11.7% subscribed to a term deposit. Metric: average precision (AP).
A leak in plain sight; the default booster against logistic regression; the learning rate with early stopping.
Notebook on Colab (in Portuguese): nothing to download; XGBoost and LightGBM preinstalled.
At every step, answer the Predict before running.
Simulations: a forest of stumps against a boosting of stumps (step 4) and more trees, no brakes (step 5).

Pace: steps 1 to 3 in 10 min; step 4 and the first simulation in 12 min; step 5 and the second in 13 min.