Machine Learning · Insper

Cross-Validation

Fundamental approach

Cross-validation (CV): scoring a recipe without depending on the luck of one split

The path of this part

Three setstrain, validation, test; why validation wears out
One split → k foldsthe lottery of one split; k-fold: k parts take turns validating; mean ± std
Honest foldsthe right splitter, preprocessing refit in every fold
The honest recipefive rules, then the lab: tune C with a grid search

Each stage answers one question: which data may judge the model? → how do we escape the luck of one split? → what must stay apart inside each fold? → what number do we report, and how?

Stage 1 · Which data may judge the model?

Three sets, three roles

The dataset split into 60% train, 20% validation and 20% test, with what each block answers

train	the model learns its parameters here	touched always
validation	chooses between models and hyperparameters	touched many times
test	simulates the future	touched once

Because you make decisions on validation, performance there drifts optimistic over time. Iterate against the test set and it silently becomes a validation set.

Why a set you choose on stops being honest

Pure noise: 20 random columns, a coin-flip target. No model can beat 50%. Train 100 trees with random hyperparameters and keep the best on validation.

Predict. The winner scores above 50% on validation. What will the same tree score on untouched data? Does the gap grow or shrink as you compare more configurations?

The validation score of the best of K models rises to 61% on pure noise while the test score stays at chance

The winner scores 61.2% on validation and 50.9% on untouched data. No model improved: you picked the one that happened to match the quirks of those 100 rows.

This is the winner's curse: the bias grows with the number of things you tried.

Stage 2 · How do we escape the luck of one split?

A single split is a lottery

Predict. Only the random_state of the split changes. How far apart can two reported accuracies be? And two means of a 5-fold CV?

Hold-out accuracy runs from 93% to 100% depending on the seed; the 5-fold CV mean only from 96.8% to 98.4%

Same model, same data, 300 seeds of an 80/20 split:

one hold-out	93.0% to 100.0%
5-fold CV mean	96.8% to 98.4%

A student reporting 100% and one reporting 93% may have written identical code.

CV greatly reduces the luck of the split; it cannot remove the luck of the sample — all folds come from the same rows.

Simulator: the luck of the split

Predict. With \(n = 80\) rows (\(n\) = number of rows), press Draw 100: which band is wider, the hold-out or the CV mean? Then press New dataset a few times: does the CV band stay on the white line?

k-fold, step by step

Cut the training data into \(k\) folds; each round trains on the other \(k-1\) and scores on the one left out; report the mean and the standard deviation of the \(k\) scores.

k-fold in code

The CV runs on X_train only: the test set stays locked away, as in the diagram.


cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(pipe, X_train, y_train, cv=cv, scoring='f1')
print(f"{scores.mean():.3f} ± {scores.std():.3f}")

Every observation validates exactly once and the model is scored on all the rows: that is where the variance reduction comes from. The price is \(k\) fits.

The spread is information too

If the folds give 0.97 · 0.96 · 0.98 · 0.71 · 0.97, the mean hides a model that fails badly on some subset. That fold deserves a look.
A recall of 0.95 ± 0.05 and one of 0.95 ± 0.005 are very different statements.

The \(k\) models are thrown away. CV scores a recipe (algorithm + hyperparameters + preprocessing); what goes to production is that recipe refit once on all the training data.

Stage 3 · What must stay apart inside each fold?

Choosing a CV scheme

Folds produced by KFold, StratifiedKFold, GroupKFold and TimeSeriesSplit on 60 rows sorted by class and group

The splitters differ in which rows may land together in a validation fold.

Which splitter to use

method	when to use	watch out for
K-Fold	regression, balanced data (with `shuffle=True`)	folds with no rows of a rare class
Stratified K-Fold	classification, especially imbalanced	only stratifies on the target
Leave-One-Out	very small datasets	\(n\) fits (\(n\) = number of rows); one row per fold: no per-fold std
TimeSeriesSplit	temporal data	early folds train on little; `gap` if the target looks ahead
Group K-Fold	grouped data (patients, users)	needs a group column

Predict. 100 rows, 8 positives, KFold shuffled into 5 folds: how often does some fold get no positive at all?

Shuffling is not enough: with 100 rows and 8 positives, shuffled KFold left some fold with no positives in 65% of the seeds. Stratification guarantees 1 or 2 in every fold.

Preprocessing restarts in every fold

Predict. If the scaler is fit on all of X before CV, will the score come out higher, lower or about the same? And if the step were a feature selector that looks at y?


# WRONG: the validation rows' statistics leak into training
X_scaled = StandardScaler().fit_transform(X)
cross_val_score(LogisticRegression(), X_scaled, y, cv=cv)

# RIGHT: each fold gets its own scaler
pipe = make_pipeline(StandardScaler(), LogisticRegression())
cross_val_score(pipe, X, y, cv=cv)

Scaling, encoding, imputation and feature selection must be refit inside each fold.
In a Pipeline, every step is fit only on the fold's training portion and applied frozen to its validation portion.

How much the mistake costs depends on what the step learns, and the advanced part measures it: a scaler changes almost nothing here; a feature selector fabricates 36 points out of nothing.

Stage 4 · What number do we report, and how?

The honest-validation recipe

Lock the test set away, with stratify for classification.
Replace the split with CV: StratifiedKFold(5, shuffle=True, random_state=...).
Build each candidate as a Pipeline, with every fitted step inside.
Report mean ± std and look at the individual folds.
Reuse a single cv object across candidates — the comparison becomes paired.

CV replaces the validation set, never the test set.

Fundamental lab · about 35 min, in pairs

Breast cancer: CV in action, steps 1 to 6

569 tumours, 30 measurements; the metric is recall on the malignant class.
Notebook on Colab (in Portuguese): nothing to download. Save a copy to your Drive.
In each step, answer the Predict before running.
Simulations: the luck of the split, measured (step 4) and the one-standard-error rule (step 6).

Pace: steps 1 to 3 in 10 min; step 4 and its simulation in 12 min; steps 5 and 6 with the simulation in 13 min.