Machine Learning · Insper

Gradient Boosting

Advanced approach

Other losses, the modern libraries, and when boosting beats the forest

The path of this part

Any loss, same looprobust losses, log-odds, AdaBoost
Tune the other knobstree size, leaves, L2, with η fixed
The modern librarieshistograms, XGBoost, LightGBM, CatBoost
Forest or boosting?a fair comparison, side by side

Each stage answers one question: what does the next tree listen to? → which knobs matter once the learning rate and early stopping are set? → what do the libraries add to the algorithm? → which ensemble should I use?

Stage 1 · what does the next tree listen to?

The loss decides what the next tree listens to

loss	pseudo-residual \(-\partial L/\partial F\)	\(F_0\)
squared	\(y - F\)	mean
absolute	\(\operatorname{sign}(y - F)\)	median
Huber	\(y - F\), clipped at \(\pm\delta\)	—
log loss	\(y - p\)	log-odds of the base rate

An outlier, \(y_6 = 40\): squared pseudo-residuals −9.33 … 28.67; absolute ones, all \(\pm 1\).

With the squared loss, the first stump spends its split on the outlier; with the absolute or Huber loss, every point pushes equally hard.

Classification: adding log-odds

The ensemble adds \(F\) (log-odds) and applies the sigmoid: \(p = 1/(1 + e^{-F})\). Pseudo-residual: \(y - p\).
Six points, 4 positive: \(F_0 = \ln 2 = 0.693\). The stump splits at \(x < 3.5\).
Each leaf gets a Newton step: \(\sum (y_i - p_i) / \sum p_i (1 - p_i) = \pm 1.5\).

\[ F_1 = 0.693 \pm 0.5 \times 1.5 \;\Rightarrow\; p = 0.486 \text{ and } 0.809 \]

By hand and scikit-learn agree to three decimals; one stump lowers the log loss from 0.637 to 0.448. Dividing by \(p(1-p)\), the curvature, keeps the step cautious where the model is unsure.

The 12 churn customers: averaging or adding stumps

The same stump, seven times. Bagging: each stump sees a bootstrap sample of \((x, y)\) and the model is the average of the probabilities. Boosting: each stump sees the residuals \(y - p\) and its leaf is added to the log-odds, × 0.5, with \(p = \sigma(F)\) — adding probabilities could leave \([0, 1]\). Predict. In boosting, does the second stump repeat the first one's question?

AdaBoost, the ancestor

Instead of residuals, it reweights the points: the misclassified gain weight, so the next stump must look at them.
A stump wrong on 3 of 10: \(\alpha = \tfrac12 \ln\frac{0.7}{0.3} = 0.424\); each misclassified point goes from 10% to 16.7%; together, half the weight.

10 features, sum of squares (ESL §10.1)	test error
one stump	45.9%
one full tree	24.5%
AdaBoost, 400 stumps	11.6%
gradient boosting, 400 stumps	11.1%

Simulator: AdaBoost on the 12 churn customers

The size of each customer is its weight. Predict. Round 1 (bill ≤ 82.5) misclassifies H and K: who grows after round 2?

Stage 2 · which knobs matter once the learning rate and early stopping are set?

The other knobs, with \(\eta\) and early stopping fixed

California housing, 5-fold CV	RMSE	trees
defaults (31 leaves)	45.8 ± 0.6	350
`max_leaf_nodes=8`	46.3 ± 0.8	736
`max_leaf_nodes=127`	45.9 ± 0.6	179
`min_samples_leaf=200`	46.4 ± 0.3	452
`l2_regularization=10`	45.5 ± 0.4	419

Once \(\eta\) and the number of trees are right, the other knobs change the error little and how many trees are needed a lot. Tuning order: moderate \(\eta\) + early stopping → tree and leaf size → subsampling and L2.

Simulator: the error barely moves, the number of trees does

Predict. Which variant leaves the noise band of the defaults? And which changes the number of trees the most?

Stage 3 · what do the libraries add to the algorithm?

The modern libraries

Histograms (LightGBM, XGBoost hist, HistGradientBoosting*): up to 255 bins per feature; native missing values and categories.
XGBoost: a penalty per tree and the second derivative; each leaf's optimal value is

\[ w_j^* = -\frac{G_j}{H_j + \lambda} \qquad \lambda = 0:\ -1.5 \quad \lambda = 1:\ -0.6 \]

LightGBM: grows leaf-wise (num_leaves), GOSS and EFB for large data.
CatBoost: encodes categories by the target mean of earlier rows, never seeing the row's own label.

With \(\lambda = 0\), \(w^*\) is the Newton step done by hand; \(\lambda\) is Ridge inside each leaf.

Simulator: the XGBoost leaf

\(w^* = -G/(H+\lambda)\); the split stays if its gain beats \(\gamma\). Predict. On the 12 customers, as \(\lambda\) grows, which leaf shrinks faster: the one with fewer customers or more?

Stage 4 · which ensemble should I use?

Forest or boosting?

R² of 16 random forest and boosting configurations

Defaults, fold by fold: boosting beats the forest on every fold, +1.6 ± 0.4 thousand of RMSE; tuned, +3.8 ± 0.4.
16 random configurations: forest from R² 0.762 to 0.826; boosting from 0.650 to 0.844.

The forest is hard to spoil; tuned boosting goes further, but a mismatched configuration ends up much worse. Early stopping takes the most dangerous knob out of the search.

Simulator: forest or boosting, point by point

Predict. What do the worst boosting configurations have in common? Click them, then switch to the budget view η × trees.

Side by side

	Random forest	Gradient boosting
trees	independent, deep	in sequence, shallow
reduces	variance (average)	bias (sum)
more trees	never hurt	end up overfitting
untuned	very good	good; bad settings can be much worse
tuned	a little better	usually the best on tabular data
for free	OOB, parallel training	early stopping, missing values, losses

Advanced lab · about 40 min, in pairs

Compare, tune and test, steps 6 to 11

Same notebook, same session: the steps use X_dev, cv, cv_eval, summary and results.
The forest, XGBoost and LightGBM on the same folds; a random search; the out-of-fold threshold; the test, once; the columns the model uses.
Notebook on Colab (in Portuguese), from step 6.
Simulations: stochastic boosting (step 7) and and the following year? (step 11).

Pace: steps 6 and 7 and the first simulation in 15 min; step 8 in 10 min; steps 9 and 10 in 8 min; step 11 and the second in 7 min.