Validation & Data Leakage

The golden rule — never evaluate on training data — sounds trivial. This lesson turns it into engineering practice: how to split data, how to validate honestly with limited data, and how to spot data leakage, the failure mode responsible for most "too good to be true" results in real projects and competitions.

Train / validation / test

flowchart LR
    D[All data] --> TR[Training set<br><small>fit models</small>]
    D --> VA[Validation set<br><small>compare models,<br>tune hyperparameters</small>]
    D --> TE[Test set<br><small>final estimate,<br>touched ONCE</small>]

• Training set — the model learns its parameters here;
• Validation set — used to choose between models and hyperparameter values. Because you make decisions on it, performance on it becomes optimistically biased over time;
• Test set — simulates the future. Used once, at the very end. If you iterate against the test set, it silently becomes a validation set and you no longer have an honest estimate.

from sklearn.model_selection import train_test_split

X_temp, X_test, y_temp, y_test = train_test_split(X, y, test_size=0.2, random_state=42,
                                                  stratify=y)   # keep class ratios
X_train, X_val, y_train, y_val = train_test_split(X_temp, y_temp, test_size=0.25,
                                                  random_state=42, stratify=y_temp)

stratify=y preserves class proportions in every split — important for imbalanced classification (see ROC-AUC & Imbalanced Data).

Cross-validation

A single validation split wastes data and gives a noisy estimate — with small datasets, the luck of the split can dominate. k-fold cross-validation (typically \(k = 5\) or \(10\)) fixes both:

1. partition the training data into \(k\) folds;
2. for each fold: train on the other \(k-1\), evaluate on it;
3. report the mean (and standard deviation!) of the \(k\) scores.

flowchart TD
    subgraph 5-fold CV
    R1["fold1 = <b>val</b> | fold2..5 = train"]
    R2["fold2 = <b>val</b> | others = train"]
    R3["..."]
    R5["fold5 = <b>val</b> | others = train"]
    end
    R1 & R2 & R3 & R5 --> M["mean ± std of 5 scores"]

from sklearn.model_selection import cross_val_score, StratifiedKFold

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
scores = cross_val_score(pipe, X_train, y_train, cv=cv, scoring='f1')
print(f"{scores.mean():.3f} ± {scores.std():.3f}")

Every observation serves as validation exactly once; the spread of scores tells you how much to trust the mean.

Choosing a CV scheme

The test set stays outside the whole procedure: cross-validation replaces the validation set, not the test set.

Preprocessing must restart in every fold

Scaling, encoding, imputation, and feature selection must be re-fit from scratch inside each fold. If you StandardScaler().fit_transform() or SelectKBest().fit() on the whole dataset before cross_val_score/GridSearchCV, the validation fold's statistics leak into training — an optimistically biased score. Passing a Pipeline makes this automatic: in every fold, all steps are fit only on that fold's training portion, then applied frozen to the validation portion. Each fold gets its own scaler, its own selected features, and its own model.

Data leakage

Leakage = information from outside the training fold sneaking into training. The model looks brilliant in evaluation and collapses in production, where the leaked information does not exist yet. The classic patterns:

1. Preprocessing before splitting

Fitting a scaler, imputer, or feature selector on all data before the split leaks test-set statistics into training. You met this in Preprocessing; the cure is structural — put every fitted step inside a Pipeline so it is re-fit within each fold.

2. Features that know the future

A column recorded after the prediction moment: predicting hospital readmission using number_of_followup_visits; predicting churn using account_closed_date is null. Flawless in the historical table, nonexistent at prediction time. Ask of every feature: "is this value available at the moment the prediction must be made?"

3. Duplicate or near-duplicate rows

The same record (or trivially perturbed copies) landing in both train and test — common after data augmentation or joins. The model is graded on questions it memorized.

4. Group leakage

Multiple rows per entity (several images per patient, several orders per customer) split randomly: the model recognizes the patient, not the disease. Use GroupKFold keyed on the entity.

5. Temporal leakage

Random shuffling of time-stamped data trains the model on the future to predict the past. Always split chronologically; validate with TimeSeriesSplit.

A leak-free experiment, end to end

The complete honest workflow, as practiced in class on the breast-cancer dataset — hold-out split, pipeline, stratified CV inside a grid search, one final fit, one final measurement:

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split, StratifiedKFold, GridSearchCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import roc_auc_score

# 1. Final hold-out split (very important!)
X, y = load_breast_cancer(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.15, random_state=42, stratify=y)

# 2. Pipeline: every fitted step inside, so CV re-fits it per fold
pipeline = Pipeline([
    ('scaler', StandardScaler()),
    ('clf', RandomForestClassifier(random_state=42)),
])

# 3. Model development: grid search with stratified CV on TRAIN only
cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
param_grid = {
    'clf__n_estimators': [100, 200, 300],
    'clf__max_depth': [5, 10, 20],
}
grid = GridSearchCV(pipeline, param_grid, cv=cv, scoring='roc_auc', n_jobs=-1)
grid.fit(X_train, y_train)

# 4. The production model: best config refit on ALL training data
final_model = grid.best_estimator_
final_model.fit(X_train, y_train)

# 5. Honest evaluation — only now, only once
proba = final_model.predict_proba(X_test)[:, 1]
print("test ROC AUC:", roc_auc_score(y_test, proba))

Exercises

1. On the iris dataset, compare K-Fold, Stratified K-Fold, and Leave-One-Out cross-validation: report mean ± std accuracy for each and explain the differences.
2. Using the BOVESPA index (Close prices via yfinance, ticker ^BVSP, 2 years): use TimeSeriesSplit for cross-validation and fit a linear regression to forecast the index 20 days ahead. Why would shuffled K-Fold be dishonest here?
3. Research SelectKBest: why must feature selection also live inside the pipeline?

Class materials

Quiz