Model Selection

You can now train several model families and validate them honestly. Model selection is the discipline of choosing among them — and among their hyperparameters — without fooling yourself. Its theoretical heart is the bias–variance trade-off; its oldest trap is regression to the mean; its workhorse tool is grid search with cross-validation.

The bias–variance trade-off

Imagine retraining your model on many different samples of training data and looking at its prediction for one fixed point \(x\). For squared error, the expected test error decomposes:

• Bias — systematic error from a model too simple to represent the truth (degree-1 fit to a sine: wrong everywhere, in the same way, on every retraining);
• Variance — sensitivity to the particular training sample (degree-15 fit: wildly different curves for each sample of 25 points);
• Noise — the part no model can remove.

Simple models: high bias, low variance → underfitting. Complex models: low bias, high variance → overfitting. Somewhere between lies the sweet spot — visible experimentally in a validation curve:

Training error (orange) falls monotonically with complexity — it cannot see overfitting. Validation error (blue) is U-shaped: it falls while complexity reduces bias, then rises as variance takes over. Select complexity at the bottom of the blue curve, never the orange one.

Knobs that move you along this curve: polynomial degree, regularization α (inverted: larger α = simpler), tree depth (Decision Trees), k in k-NN (inverted), number of features.

Drive the trade-off yourself — slide from degree 0 (pure bias) to 12 (pure variance) and watch the two errors part ways:

Learning curves: is more data worth it?

Plot train and validation scores versus training-set size:

• curves converge at a low score → high bias: more data won't help; add capacity or features;
• large persistent gap → high variance: more data (or regularization) will help.

Regression to the mean

Galton (1886) noticed children of very tall parents are usually closer to average height. The general phenomenon: extreme observations are partly luck, and luck does not repeat. Whenever performance = skill + noise, the top performer of round one tends to fall back in round two — with no causal explanation needed.

Why this haunts model selection: compare 50 model configurations by validation score and pick the winner. The winner won partly by being good and partly by being lucky on that validation data. Its reported score is optimistically biased — expect it to regress when retested. Concretely:

• the more configurations you try, the more inflated the best validation score becomes;
• this is why the untouched test set exists: it gives the winner one honest, luck-free measurement;
• the same effect explains why last year's Kaggle winner underperforms on fresh data, and why the "best fund manager of 2025" disappoints in 2026.

Grid search with cross-validation

GridSearchCV automates honest hyperparameter selection: for every combination in a grid, run k-fold CV; pick the best mean score; refit on all training data.

from sklearn.model_selection import GridSearchCV
from sklearn.pipeline import Pipeline

pipe = Pipeline([('preprocess', preprocess), ('model', Ridge())])

param_grid = {
    'model__alpha': [0.01, 0.1, 1, 10, 100],
    'preprocess__num__imputer__strategy': ['mean', 'median'],
}

search = GridSearchCV(pipe, param_grid, cv=5,
                      scoring='neg_root_mean_squared_error', n_jobs=-1)
search.fit(X_train, y_train)

search.best_params_       # winning combination
search.best_score_        # its mean CV score (optimistic — see above!)
search.best_estimator_    # pipeline refit on ALL training data
search.score(X_test, y_test)   # the honest number

Design notes:

• Search over the pipeline, not the bare model — preprocessing choices are hyperparameters too, and CV inside the pipeline stays leak-free;
• Grid cost is multiplicative (5 α's × 2 strategies × 5 folds = 50 fits) — for large spaces use RandomizedSearchCV, which samples combinations and usually finds near-optimal settings much faster (Bergstra & Bengio, 2012), or the smarter strategies of AutoML;
• Choose scoring to match the problem — F1 or ROC-AUC for imbalanced classification, not accuracy;
• Prefer log-spaced grids for scale parameters (α, C): [0.01, 0.1, 1, 10, 100].

Occam's razor, operationalized

When two configurations score within noise of each other (compare their CV standard deviations), prefer the simpler one — fewer features, stronger regularization, shallower trees. Simpler models are cheaper to serve, easier to explain, and more robust to drift.

Class materials

Quiz