Train/Val/Test Split

Train / Validation / Test Split

The split is not a preparation step. It is the experimental design of the whole project: it decides what question your final number answers, and how much you are entitled to believe it.

Two things go wrong, and they are opposites. Either the split lets information across — in which case the number is too high, and leakage is the chapter about that. Or the split is honest but too small, or used too often, in which case the number is too noisy to support the decision you are making with it. This page is mostly about the second, because almost nobody measures it.

1. Splitting what, exactly

train_test_split splits rows. Whether rows are the right unit is a question about your data, and getting it wrong is the most common real-world split bug.

from sklearn.model_selection import train_test_split, GroupShuffleSplit, TimeSeriesSplit

# classification: stratify, always
X_tr, X_te, y_tr, y_te = train_test_split(X, y, test_size=.2, stratify=y, random_state=42)

# repeated measurements: the unit is the patient, not the visit
train, test = next(GroupShuffleSplit(n_splits=1, test_size=.2, random_state=42)
                   .split(X, y, groups=patient_id))

# ordered data: the test set is the future
cut = '2026-01-01'
train, test = df[df.index < cut], df[df.index >= cut]

The question to ask before choosing is not "how do I split this?" but "what is the model supposed to generalize to?" A new patient, a new user, next month, a new hospital. Whatever that unit is, it is the unit the split must keep whole — because the test set is a rehearsal of deployment, and in deployment that unit is always new.

2. How much do you actually know?

A test accuracy is a proportion estimated from a finite number of rows, so it comes with a confidence interval like any other estimate. Almost nobody computes it, and it is usually much wider than the three decimal places people quote.

The interval shown is the Wilson score interval, which is the right one for a proportion — the textbook \(p \pm 1.96\sqrt{p(1-p)/n}\) runs off the end of the scale near 0 and 1, which is exactly where a decent classifier lives.

Two consequences worth internalizing:

• A 200-row test set cannot tell 0.90 from 0.94. The interval at \(n = 200\), \(p = 0.90\) runs from 0.851 to 0.934. Two models inside that window are not better and worse; they are indistinguishable.
• Precision costs rows quadratically. The width falls like \(1/\sqrt{n}\), so halving the window means four times the test set. This is why the percentage split changes with dataset size: at 10 million rows, 1% is 100 000 test rows, and a further 9% buys almost nothing in precision while costing real training data.

3. Cross-validation

When a single split is too noisy, use every row for validation in turn.

5-fold:   [VAL][TRN][TRN][TRN][TRN]     each row is validated exactly once,
          [TRN][VAL][TRN][TRN][TRN]     and trained on four times
          [TRN][TRN][VAL][TRN][TRN]
          [TRN][TRN][TRN][VAL][TRN]     report mean ± sd across the folds
          [TRN][TRN][TRN][TRN][VAL]

from sklearn.model_selection import cross_val_score, StratifiedKFold

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(pipeline, X, y, cv=cv, scoring='f1_macro')
print(f"{scores.mean():.3f} ± {scores.std():.3f}")

Two things people get wrong here:

1. Cross-validate the Pipeline, not the model. If the scaler, imputer or feature selector is fitted outside, all five folds are contaminated at once. The leakage chapter measures what that costs.
2. A cross-validated score is still a validation score. If you compared twenty configurations with it, the best of the twenty is optimistic — see section 4. Keep a test set that the cross-validation never touched, or use nested CV: an inner loop to choose, an outer loop to measure.

from sklearn.model_selection import GridSearchCV, cross_val_score

search = GridSearchCV(pipeline, grid, cv=StratifiedKFold(5))     # inner: chooses
honest = cross_val_score(search, X, y, cv=StratifiedKFold(5))    # outer: measures

4. The golden rule, and what breaking it costs

The reason people break it is that the cost feels abstract. It is not, and step 2 of the lab puts a number on it: choosing the best of forty genuinely equivalent models on a 200-row set inflates the reported accuracy by 1.4 points of pure noise.

If you must iterate — and you must — the structure that lets you do it honestly is:

• a validation set or an inner CV loop, which you may consume freely;
• a test set opened once, at the end, to produce the number you publish;
• and if you open it and are unhappy, the honest move is to report it anyway and note how many configurations were tried.

Lab: how much is a split estimate worth?

Step 1 — the same model, 120 different seeds

Nothing changes between the runs except which rows land in the test set.

"""The same model, the same data, 120 different values of random_state.

Nothing changes between the runs except which rows happen to land in the test
set. The spread of the resulting accuracies is the precision of the number you
would have reported from a single split — and it is far wider than the two
decimal places everybody quotes.

The last row is not measured but predicted: a test accuracy is a proportion
estimated from n_test rows, so its standard error is sqrt(p(1-p)/n_test).
Compare it to the standard deviation measured on the row above.

Printed as a markdown table, in identifiers only, so one artifact serves both
the English and the Portuguese page.
"""

import numpy as np
from sklearn.model_selection import StratifiedKFold, cross_val_score, train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

N, DIM, REPEATS = 600, 8, 120

rng = np.random.default_rng(1)
direction = rng.normal(size=DIM)
X = rng.normal(size=(N, DIM))
y = ((X @ direction + 1.1 * rng.normal(size=N)) > 0).astype(int)

model = lambda: make_pipeline(
    StandardScaler(), MLPClassifier(hidden_layer_sizes=(16,), max_iter=2000, random_state=0)
)

single = []
for seed in range(REPEATS):                 # only the seed changes, nothing else
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size=0.2, random_state=seed, stratify=y
    )
    single.append(model().fit(X_train, y_train).score(X_test, y_test))
single = np.array(single)

folds = []
for seed in range(REPEATS // 5):            # the same budget of fits, spent differently
    cv = StratifiedKFold(5, shuffle=True, random_state=seed)
    folds.append(cross_val_score(model(), X, y, cv=cv).mean())
folds = np.array(folds)

print("| `estimator` | `mean` | `sd` | `min` | `max` | `spread` |")
print("|---|---:|---:|---:|---:|---:|")
for name, v in (("single_80_20", single), ("mean_of_5_folds", folds)):
    print(f"| `{name}` | {v.mean():.3f} | **{v.std():.3f}** | {v.min():.3f} | {v.max():.3f} "
          f"| **{v.max() - v.min():.3f}** |")

n_test = int(0.2 * N)
p = single.mean()
print(f"| `binomial_se(n={n_test})` | — | **{np.sqrt(p * (1 - p) / n_test):.3f}** | — | — | — |")

The same model, the same data, the same code. The accuracy from a single 80/20 split ranges from 0.717 to 0.875 — a spread of 15.8 points, entirely determined by random_state. Had you run this once and reported it, the number would have been anywhere in that window, and you would have had no way to know which.

The middle rows explain the fix and the cause. Averaging five folds instead of taking one split cuts the standard deviation from 0.032 to 0.006 — for the same total number of fits, because each fold trains on 80% too. And the last row is the theory: a proportion measured on 120 rows has a standard error of \(\sqrt{p(1-p)/n} = 0.037\), which is the 0.032 that was measured. Nothing unusual happened; this is just what a 120-row estimate is worth.

Step 2 — what picking the winner costs

Forty candidates, all the same architecture, differing only in the random seed. The standard deviation of their true accuracies is 0.0012 — they are, to any useful precision, the same model. So every difference between them on a 200-row selection set is noise.

"""Choosing the best of k on a held-out set, and what the winner is worth.

Forty candidates, all the same architecture, differing only in the seed — so
they are genuinely equivalent, and the standard deviation of their true
accuracies is about a thousandth. Any difference between them on a 200-row
selection set is therefore noise, and nothing else.

Picking the highest of k noisy scores and reporting it is picking the maximum
of k draws from noise. The `optimism` column is how much of the reported number
was never there: the difference between what the winner scored on the set that
chose it and what it is actually worth, averaged over 400 independent draws of
that set so that no single lucky subset can be blamed.

Printed as a markdown table, in identifiers only, so one artifact serves both
the English and the Portuguese page.
"""

import numpy as np
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

N, DIM, TRAIN = 12000, 8, 2000
CANDIDATES, SELECTION_ROWS, REPEATS = 40, 200, 400

rng = np.random.default_rng(2)
direction = rng.normal(size=DIM)
X = rng.normal(size=(N, DIM))
y = ((X @ direction + 1.1 * rng.normal(size=N)) > 0).astype(int)

fitted = [
    make_pipeline(StandardScaler(),
                  MLPClassifier(hidden_layer_sizes=(16,), max_iter=1500, random_state=seed))
    .fit(X[:TRAIN], y[:TRAIN])
    for seed in range(CANDIDATES)
]

# every candidate scored row by row on 10 000 untouched rows: the truth to compare against
correct = np.array([m.predict(X[TRAIN:]) == y[TRAIN:] for m in fitted])
truth = correct.mean(axis=1)

draw = np.random.default_rng(0)
print(f"| `candidates_tried` | `reported` | `actually_worth` | `optimism` |")
print("|---:|---:|---:|---:|")
for k in (1, 2, 5, 10, 20, 40):
    reported, actual = [], []
    for _ in range(REPEATS):
        rows = draw.choice(correct.shape[1], SELECTION_ROWS, replace=False)
        scores = correct[:k][:, rows].mean(axis=1)
        winner = int(np.argmax(scores))
        reported.append(scores[winner])
        actual.append(truth[:k][winner])
    gap = np.mean(reported) - np.mean(actual)
    print(f"| {k} | {np.mean(reported):.4f} | {np.mean(actual):.4f} | **{gap:+.4f}** |")

print()
print(f"| `sd_of_true_accuracy_across_candidates` |")
print("|---:|")
print(f"| {truth.std():.4f} |")

Read the optimism column down the page. Evaluating one model gives +0.0005 — an unbiased estimate, which is what a held-out set is supposed to provide. Trying ten and reporting the best gives +0.0101. Trying forty gives +0.0140.

None of that is improvement. The forty models are equivalent by construction, so the reported gain is the maximum of forty noisy draws, and the maximum of noise is above the mean of noise. The bias grows with how many things you tried, and it is invisible in the reported number — which is precisely why "how many configurations did you try?" is a fair question to ask of any result, including your own.