Machine Learning · Insper

Model Selection

Advanced approach

Choosing among many models without being fooled by luck

The path of this part

A common scalethe effective number of parameters
Regression to the meanwhy the winner's score falls when measured again
Compare familiesone search over all families, no free lunch, Occam's razor
The whole workflowfive steps, a one-way door before the test, then the lab

Each stage answers one question: how do we compare complexity across families? → why does the winner do worse next time? → which family wins, and how do we break a tie? → where does the test set come in?

Stage 1 · How do we compare complexity across families?

A common scale: effective parameters

For models whose prediction is a weighted sum of the training labels: add up the weight each point gives to its own label.

\(n\): training points · \(p\): columns · \(k\): neighbors in k-nearest neighbors (k-NN) · \(\alpha\): the Ridge penalty.

model	effective parameters	with \(n = 30\)
least squares, \(p\) columns	\(p\)
k-NN	\(n / k\)	\(k = 30\): 1 · \(k = 3\): 10 · \(k = 1\): 30
Ridge	falls from \(p\) toward 1 (the unpenalized intercept) as \(\alpha\) grows	continuous

A penalty turns the staircase of degrees into a ramp: that is why it is such a convenient knob. \(k = 1\) spends one parameter per point: memorization.

Simulator: three knobs, one scale

Degree, \(k\) and \(\alpha\) on the same 30-point sample, on the effective-parameters axis. Predict. Do the three bottoms fall close together?

Stage 2 · Why does the winner do worse next time?

Regression to the mean

Exam 1 against exam 2 for 10,000 students

Score = ability (standard deviation, sd, 8) + luck of the day (its sd in the first column). Reliability: \(r = \frac{\sigma^2_\text{ability}}{\sigma^2_\text{ability} + \sigma^2_\text{luck}} = \frac{8^2}{8^2 + \text{luck}^2}\).

Predict. The top 10% on exam 1: do they keep their score on exam 2? And when luck grows?

luck sd	\(r\)	exam 1	exam 2
2	0.94	84.4	83.5
6	0.64	87.6	80.9
16	0.20	101.4	76.1

Expected on exam 2: mean + \(r \times\) lead. Luck sd 6: lead \(= 87.6 - 70 = 17.6\) and \(r = 64/100 = 0.64\), so \(70 + 0.64 \times 17.6 \approx 81.3\) (simulated: 80.9). Only the part of the lead that is ability repeats.

In model selection

A configuration is a student; its future performance is the ability; its cross-validation (CV) score is an exam with luck (the folds, the sample).
Near the top of a grid, configurations barely differ (little ability) while the CV noise stays the same: low reliability.
More candidates → a more extreme winner → a luckier one. On pure noise, the best of 100 fell from 61.2% to 50.9%.

Never report the score that chose the winner. Measure again on data that played no part in the choice: the locked test set or nested CV. New folds on the same rows only show how much of the ranking is fold luck.

Simulator: regression to the mean, exact

The top 10% come back to \(70 + r \cdot D\). Predict. With \(\sigma = 0\), how far do they fall? And with \(\sigma = 16\)?

Stage 3 · Which family wins, and how do we break a tie?

The family as a hyperparameter


pipe = Pipeline([('scale', StandardScaler()), ('model', LogisticRegression())])
param_grid = [                                     # a list of grids: one per family
    {'model': [LogisticRegression(max_iter=5000)], 'model__C': np.logspace(-3, 3, 7)},
    {'model': [KNeighborsClassifier()], 'model__n_neighbors': [1, 3, 5, 9, 15, 25, 51]},
    {'model': [SVC()], 'model__C': [0.1, 1, 10, 100], 'model__gamma': ['scale', 0.01, 0.1]},
    {'model': [DecisionTreeClassifier(random_state=0)], 'scale': ['passthrough'],
     'model__max_depth': [1, 2, 3, 4, 6, 8, 12, None]},
    {'model': [RandomForestClassifier(n_estimators=200, random_state=0)], 'scale': ['passthrough'],
     'model__max_features': ['sqrt', 0.5], 'model__min_samples_leaf': [1, 5]},
    {'model': [HistGradientBoostingClassifier(random_state=0)], 'scale': ['passthrough'],
     'model__learning_rate': [0.05, 0.1], 'model__max_leaf_nodes': [7, 31]},
]

Radar returns: 42 configurations, 6 families, one search. Support vector machine (SVM) 0.946 · forest 0.943 · boosting 0.940 (a three-way tie) · tree 0.897 · logistic 0.883 · k-NN 0.866.

The model slot takes different estimators; 'passthrough' switches a step off. The same cv for all of them: they can be compared fold by fold.

No free lunch

dataset	logistic	k-NN	SVM	tree	forest	boosting
ionosphere (real)	0.883	0.866	0.946	0.897	0.943	0.940
breast cancer (real)	0.979	0.965	0.979	0.930	0.967	0.972
credit-g (real)	0.761	0.735	0.760	0.719	0.764	0.771
linear, 40-d	0.820	0.710	0.805	0.725	0.735	0.720
two moons	0.853	0.900	0.895	0.887	0.893	0.895
axis-aligned rule	0.640	0.636	0.686	0.938	0.948	0.944

Every family is at or tied for the top somewhere and loses badly somewhere (Wolpert, 1996). The ensembles are the safest bet (first or tied on 5 of 6) but lose 0.085–0.100 on the linear problem. Each family bets on a kind of structure — its inductive bias.

What each family bets on

Logistic regression

the boundary is a hyperplane

k-NN

nearby points share labels

SVM with a radial basis function (RBF) kernel

the boundary is smooth

Tree, forest, boosting

a few thresholds on a few features; the forest averages away the variance of that bet, boosting the bias

A family wins when its bet matches the problem. You cannot know that in advance: that is why model selection compares families, on the same folds.

Simulator: no free lunch

Framed = within 1 fold sd of the winner. Predict. On how many datasets are the trees (forest or boosting) on top or tied with it?

Three meanings of "bias"

The same word, three ideas of this lesson.

	what it means	what to do
bias (bias–variance)	a family's systematic error, on average over training sets	more capacity, better features
inductive bias	a family's bet on the shape of the problem	pick the family whose bet matches
selection bias	the winner of a search has an optimistic score	measure again on data that did not choose

Occam's razor, operationalized

Breast cancer: logistic and SVM, both 0.979, boosting 0.007 behind. Credit-g: boosting 0.771, logistic 0.761, one fold standard deviation apart.
On a tie within the noise, keep the simpler one: 31 numbers and a dot product, against a set of support vectors and a kernel, or hundreds of trees.
Why? A more flexible model has noisier CV scores, so its narrow win is more likely selection luck; the simpler one also depends less on the sample, holds up better when the data change and is cheaper to run and explain.

"Simpler" is not "fewer lines of code": it is fewer effective parameters, stronger regularization, fewer features.

Stage 4 · Where does the test set come in?

The whole workflow

Five model-selection stages, with a one-way door before the test

Everything left of the door uses only the training rows. Testing, looking at the number and going back to "improve" turns the test set into a validation set.

Simulator: the one-way door

42 configurations of similar accuracy, noisy scores. Predict. Is the winner's validation score above, below or equal to its true accuracy? And if you pick by the test?

Advanced lab · about 30 min, in pairs

Select and test, steps 7 to 9

Same notebook, same session: the steps use pre, cv, rmse and candidates from steps 1 to 6.
One search over five families: Ridge, k-NN, the tree, a random forest and a gradient boosting (150 fits, one to two minutes); reshuffle the folds to see how much of the ranking is luck; open the test set once.
Notebook on Colab (in Portuguese), from step 7.
Simulations: the two messages, fold by fold (step 7) and how precise is the test score? (step 9).

Pace: step 7 and the first simulation in 12 min; step 8 in 8 min; step 9 and the second simulation in 10 min.