Model Selection

The path of this part

A worked example, by hand, at \(x = 0.25\)

Each training set fits a polynomial and predicts at \(x = 0.25\), where the truth is 1. Bias \(= \overline{\hat f} - f(0.25)\); variance \(= \tfrac{1}{k}\sum_i (\hat f_i - \overline{\hat f})^2\). Predict. Which degree is right on average, and which one spreads more?

Two ways to be wrong

Truth: \(\sin(2\pi x)\). Each curve is a polynomial fitted to a different 30-point training set.

The dartboard: bias and variance

The decomposition

At a fixed \(x\): the truth \(f(x)\); a new observation \(y = f(x) + \varepsilon\), noise \(\varepsilon\) with mean 0 and variance \(\sigma^2\); the prediction \(\hat f(x)\), random because it depends on the training set; \(\mathbb{E}\): the average over training sets and noise.

1. \(y - \hat f = (f - \hat f) + \varepsilon\); the new \(\varepsilon\) has mean 0 and is independent of the training set, so the cross term vanishes: \(\mathbb{E}[(y - \hat f)^2] = \mathbb{E}[(f - \hat f)^2] + \sigma^2\).
2. For any random \(Z\): \(\mathbb{E}[Z^2] = (\mathbb{E}[Z])^2 + \operatorname{Var}(Z)\). With \(Z = f - \hat f\): bias² + variance.

Bias: the family's best attempt is wrong on average (a line cannot follow a sine). Variance: the price of listening to the sample. Noise: the part of \(y\) that does not depend on \(x\); here \(\sigma^2 = 0.3^2 = 0.09\), and nobody goes below it.

Measured with 500 training sets

Predict. 500 training sets of 30 points per degree. At which degree is the test mean squared error (MSE) lowest? And what happens to the variance at degree 12?

• The columns add up by construction (the decomposition is an identity): the sum matches the error measured on new data up to the fresh noise (difference ≤ 0.004).
• Bias falls to zero; variance rises slowly, then violently. Read against the degree, the sum is U-shaped: high on the left (bias), lowest at degree 3, high on the right (variance). The large variances are orders of magnitude: a handful of the 500 samples produce most of them.

Three families, the same U shape

• The U is the green curve, bias² + variance + noise (the expected test error), against complexity, which grows to the right (vertical axis on a log scale). It falls while bias drops faster than variance rises, and turns back up once variance dominates. Bottoms: degree 3 (0.113), \(k = 3\) (0.142) and depth 3 (0.166); the tree's U is shallow.
• k-nearest neighbors (k-NN) with \(k = 30\) (every point) is the degree-0 polynomial; in k-NN, small \(k\) is the complex model.
• 1-NN copies one noisy label: variance ≈ \(\sigma^2\) (the noise variance, 0.09), test error ≈ \(2\sigma^2\) (measured: 0.195).

Simulator: underfit ↔ overfit, live

Predict. Slide the degree from 0 (pure bias) to 12 (pure variance). Which error only falls, and which one turns back up?

Validation curve: the U without knowing the truth

• Training only falls: 0.267 → 0.046. Choosing by it always picks the most complex model.
• On the left: both high and close — bias.
• On the right: training falls, validation climbs to 1.748 — variance.
• Bottom at degree 3 (0.114): the same as the 500-sample simulation.

Learning curve: is more data worth it?

Predict. More and more training points: does the error of degree 1 keep falling? And that of degree 9?

• Median of 300 training sets per size.
• Degree 1: training and test meet near 0.29 and stay there. More data does not help.
• Degree 9: test error 5.6 with 12 points, at the noise floor by about 200. With its coefficients fixed, data closes the gap.

The diagnosis

Four tools that get mixed up

What sets them apart is what is varied and what stays fixed.

Every family has a knob

A knob is the hyperparameter that moves the model along the complexity axis of the U: to the left, more bias (underfitting); to the right, more variance (overfitting). High bias asks for a turn to the right; high variance, to the left. Predict. To make each model more complex, which way do you turn each knob?

GridSearchCV: the validation curve that chooses by itself

from sklearn.model_selection import GridSearchCV, RepeatedKFold

# 1. the model, with the knob still loose
model = make_pipeline(PolynomialFeatures(), StandardScaler(),
                      LinearRegression())

# 2. the knob values to try
#    (step name + __ + hyperparameter)
grid = {"polynomialfeatures__degree": [1, 2, 3, 4, 5, 6, 7, 8, 9, 10]}

# 3. how to measure each value: 5 folds × 20 repeats (slide 10's cv)
cv = RepeatedKFold(n_splits=5, n_repeats=20, random_state=0)
search = GridSearchCV(model, grid, cv=cv,
                      scoring="neg_mean_squared_error")

# 4. measure the 10 degrees, pick the lowest error
#    and refit the winner on the 30 rows
search.fit(X, y)
search.best_params_      # {'polynomialfeatures__degree': 3}
    

Summary

1. Test error = bias² + variance + noise. The noise is a floor.
2. Bias: training ≈ validation, both high. Variance: training ≪ validation.
3. A validation curve varies complexity; a learning curve varies the data.
4. For a model of fixed capacity, more data cures variance and not bias; more capacity only cures bias.
5. Every model has a complexity knob, and some turn backwards.

The typical patterns

On the training and validation error curves (here, low is good):

"High" is relative: without knowing \(\sigma^2\) (the noise variance), compare with a baseline (predicting the mean, a simple model) or with the error the business can accept.

House prices: diagnose and treat, steps 1 to 6

• California housing: 20,640 districts, 9 features, target = median house value.
• Three families (linear, k-NN, tree): which suffers from bias and which from variance? Would more data help? Which knob to turn?
• Notebook on Colab (in Portuguese): nothing to download. Save a copy to your Drive.
• At every step, answer the Predict before running.
• Simulations: variance, up close (step 3) and alpha as a continuous knob (step 6).

Pace: steps 1 to 3 and the first simulation in 15 min; steps 4 to 6 and the second in 25 min.