Data Leakage

Data Leakage

A model is a promise about the future: given what I will know at the moment of the decision, here is what I predict. Data leakage is the moment that promise quietly stops being true — when something the model was trained on, or evaluated against, would not have been available at the moment of the decision.

The symptom is always the same, and it is the reason leakage is so dangerous: the numbers go up. Validation accuracy climbs, the loss curve looks beautiful, the team celebrates. Nothing in the training log says anything is wrong, because as far as the optimizer is concerned nothing is wrong — it was handed the answer and it used it. The failure only appears later, in production, where the answer is no longer in the box.

Everything below is a different way of getting that question wrong.

Where leakage comes from

Leakage is not one bug. It is four, and they enter at four different places in the workflow, which is why fixing one of them does not protect you from the others.

And a fifth, which is not a bug in the code but a bug in the person: the test set is consumed a little bit every time you look at it. More on that at the end.

1. Leakage through a column

The most literal kind: a feature that exists because the answer exists.

The task: predict whether a patient will be prescribed antibiotic X.

The model learns took_antibiotic_x == True → prescribed == True. It is right almost always, it is worthless always, and it will score 0.99 on any validation set you build.

The useful mental move is to draw the timeline. Put the prediction moment on it as a vertical line, then place every feature on one side or the other. A feature to the right of the line is a leak. A feature that straddles the line — n_notes_in_chart, total_spend, anything that accumulates — is the dangerous case, because it is partly legitimate and so it survives review.

2. Leakage through the split

Now the features are clean and the code is clean, and the model still cheats — because train_test_split shuffles rows, and rows are not independent.

One patient, eight visits. One user, two hundred clicks. One song, forty spectrogram windows. One document, its own near-duplicate scraped twice. Shuffle rows, and the same patient is in training and in test. The model no longer has to learn medicine; it only has to recognize the patient, whose answer it was already given.

This is the single most common leak in real projects, and the hardest to see, because nothing in the code looks wrong.

# ❌ the same patient can land on both sides
X_train, X_test = train_test_split(X, test_size=0.2, random_state=42)

# ✅ a patient belongs to exactly one side
from sklearn.model_selection import GroupShuffleSplit, GroupKFold
splitter = GroupShuffleSplit(n_splits=1, test_size=0.2, random_state=42)
train, test = next(splitter.split(X, y, groups=patient_id))

The question to ask is not "are my rows independent?" but "what is the unit I am actually generalizing to?" If the model is meant to work on a new patient, then the patient is the unit, and the patient is the group. If it is meant to work on a new visit of a known patient, a row split is fine — but that is a very different product, and it is almost never the one that was promised.

Near-duplicates count too. Two photographs taken one second apart, a record inserted twice by a broken ETL, an augmented copy generated before the split — each is a group of size two, and each hands the answer across the split just as effectively.

3. Leakage through time

Time turns both of the previous problems up.

The split. In a time series, a random split trains on Wednesday and Friday to predict Thursday. The model is interpolating inside a window it has already seen — which is not the task. The task is always extrapolation: everything you train on comes strictly before everything you test on.

        train                    │  test
──●──●──●──●──●──●──●──●──●──●───│──○──○──○──○──○──►
                                 │
                      no sample from the right
                      of this line may touch the left

# ❌ random split on a time series
X_train, X_test = train_test_split(X, y, test_size=0.2, random_state=42)

# ✅ the split is a date, not a shuffle
from sklearn.model_selection import TimeSeriesSplit
cv = TimeSeriesSplit(n_splits=5)              # each fold trains on the past only

The feature. This one survives the correct split, which is what makes it nasty. Any window that is centred on the current instant reaches into the future:

# ❌ a window centred on t spans t-1, t and t+1
df['ma3'] = df['r'].rolling(3, center=True).mean()

# ❌ the same leak, wearing a different hat
df['z'] = (df['r'] - df['r'].mean()) / df['r'].std()   # mean over the whole series
df['rank'] = df['r'].rank(pct=True)                    # rank against the whole series
df = df.fillna(df.median())                            # median of the whole series

# ✅ everything is computed from the past, and only the past
df['ma3'] = df['r'].rolling(3).mean()
df['ma3'] = df['ma3'].shift(1)         # and not even today, if today closes after the decision

There is also a gap to respect. If you decide at 09:00 and the label resolves at 17:00, a feature that becomes available at 12:00 is a leak even though it is in the past relative to the label. The clock that matters is the decision's, never the label's.

4. Leakage through the pipeline

The textbook case: a preprocessing step fitted on data that includes the test rows. StandardScaler learns a mean and a standard deviation; SimpleImputer learns a median; TargetEncoder learns a per-category average of the label. Fit any of them before the split and those statistics carry information out of the test set and into the model.

The panel below shows, row by row, which rows each step actually touched. The distinction that matters is not train vs test — it is fitted vs merely used.

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)                      # mean and std over every row
X_train, X_test = train_test_split(X_scaled, test_size=.2)

X_train, X_test = train_test_split(X, test_size=.2)
scaler = StandardScaler()
X_train = scaler.fit_transform(X_train)                 # fit on train
X_test  = scaler.transform(X_test)                      # transform only

from sklearn.pipeline import Pipeline

pipe = Pipeline([('scaler', StandardScaler()),
                 ('model',  LogisticRegression())])
cross_val_score(pipe, X, y, cv=5)     # fit is redone inside every fold

The third tab is the one that matters. Doing it by hand is correct exactly once, for a single split; the moment you cross-validate, or tune hyperparameters, or bootstrap, the hand-written version leaks again because the scaler is only fitted once while the folds change five times. A Pipeline is not tidiness — it is the only version of the rule that survives being cross-validated.

Anything with a fit method belongs inside it: scalers, imputers, encoders, PCA, feature selectors, resamplers, discretizers.

5. Leakage through the analyst

The last one has no line of code to blame. Any decision made while looking at the test set moves information from it into the model, through you.

• Feature selection on the full dataset. Ranking 5 000 columns by their correlation with the target, keeping the best 20, then cross-validating. The filter saw every label; the folds are contaminated before they exist.
• Hyperparameter tuning on the test set. Try thirty configurations, report the best. The best of thirty is the maximum of thirty noisy numbers, and the maximum of noise is biased upward.
• Looking again. A test set used twice is not quite a test set any more. Used thirty times, it is a training set with extra steps.

The panel is the first of these, measured. There is nothing in the data — the features are gaussian noise, the label is a coin toss, and the true accuracy is exactly \(0.50\). Watch what the filter manufactures anyway:

The defence is structural, not moral: you cannot remember not to peek. Put every decision inside the resampling loop — selection as a Pipeline step, tuning inside a GridSearchCV that is itself cross-validated (nested CV) — and keep a final holdout that is opened once, at the end, to produce the number you will publish.

The checklist

Lab: four leaks, measured

Four short experiments. Each one runs the wrong pipeline and the right one on the same data and prints both scores, so the leak stops being a warning and becomes a number. Everything below is reproducible with numpy and scikit-learn alone — clone the scripts and change the constants.

Step 1 — selection before the split

The data is pure noise: X is gaussian, y is a fair coin, and the true accuracy of any model is 0.500. The only difference between the two rows is where the feature filter runs.

"""Feature selection before the split: accuracy manufactured out of pure noise.

X is gaussian noise and y is a fair coin, so there is nothing to learn and the
true accuracy of any model is 0.50. Picking the k columns most correlated with
y *before* the folds are cut lets the selector read the label of every row,
including the rows that will later be used to score the model.

Printed as a markdown table, in identifiers only, so the same artifact serves
the English and the Portuguese page.
"""

import numpy as np
from sklearn.feature_selection import SelectKBest, f_classif
from sklearn.linear_model import LogisticRegression
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.pipeline import Pipeline

N, P, K, FOLDS = 100, 5000, 20, 5

rng = np.random.default_rng(0)
X = rng.normal(size=(N, P))          # noise: no column carries any information
y = rng.integers(0, 2, size=N)       # a fair coin

cv = StratifiedKFold(n_splits=FOLDS, shuffle=True, random_state=0)
clf = lambda: LogisticRegression(max_iter=1000)

# WRONG — the selector sees y for all 100 rows, including the held-out ones
X_selected = SelectKBest(f_classif, k=K).fit_transform(X, y)
leaky = cross_val_score(clf(), X_selected, y, cv=cv).mean()

# RIGHT — selection is a pipeline step, redone inside every fold
honest = cross_val_score(
    Pipeline([("sel", SelectKBest(f_classif, k=K)), ("clf", clf())]), X, y, cv=cv
).mean()

print(f"| `n={N}, p={P}, k={K}, cv={FOLDS}` | `acc` |")
print("|---|---:|")
print(f"| `select_then_split` | **{leaky:.3f}** |")
print(f"| `split_then_select` | **{honest:.3f}** |")
print(f"| `chance_level` | {0.5:.3f} |")

Ranking 5 000 noise columns against 100 labels and keeping the 20 that happen to line up best produces 0.870 — thirty-seven points of accuracy conjured out of a random number generator. With \(5\,000\) columns and \(100\) rows, some column will correlate with the coin by luck; the filter's entire job is to find it. Moving the same filter inside the fold returns the honest 0.500.

Step 2 — the same patient on both sides

120 patients, eight visits each. The visits of one patient look alike, and only one coordinate of that resemblance has anything to do with the diagnosis.

"""Group leakage: the same patient on both sides of the split.

Every patient contributes eight visits, and the visits of one patient look
almost alike — they share a fingerprint. Only one coordinate of that
fingerprint is related to the label, so an honest model can do better than
chance and no more. A shuffled split puts some visits of each patient in train
and the rest in test; the network then stops classifying and starts
recognizing the patient whose answer it has already been shown.

Printed as a markdown table, in identifiers only, so one artifact serves both
the English and the Portuguese page.
"""

import numpy as np
from sklearn.model_selection import GroupKFold, StratifiedKFold, cross_val_score
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

PATIENTS, VISITS, DIM, FOLDS = 120, 8, 12, 5

rng = np.random.default_rng(1)
fingerprint = rng.normal(size=(PATIENTS, DIM))          # who the patient is
# only coordinate 0 is related to the diagnosis, and only loosely
label = (fingerprint[:, 0] + 0.7 * rng.normal(size=PATIENTS) > 0).astype(int)

patient = np.repeat(np.arange(PATIENTS), VISITS)        # the group id
X = fingerprint[patient] + 0.20 * rng.normal(size=(PATIENTS * VISITS, DIM))
y = label[patient]

model = lambda: make_pipeline(
    StandardScaler(), MLPClassifier((32,), max_iter=2000, random_state=0)
)

# WRONG — visits are shuffled, so a patient sits in train and in test at once
leaky = cross_val_score(
    model(), X, y, cv=StratifiedKFold(FOLDS, shuffle=True, random_state=0)
).mean()

# RIGHT — a patient belongs to exactly one fold
honest = cross_val_score(
    model(), X, y, cv=GroupKFold(FOLDS), groups=patient
).mean()

print(f"| `{PATIENTS} patients x {VISITS} visits = {len(y)} rows, cv={FOLDS}` | `acc` |")
print("|---|---:|")
print(f"| `StratifiedKFold(shuffle=True)` | **{leaky:.3f}** |")
print(f"| `GroupKFold(groups=patient)` | **{honest:.3f}** |")
print(f"| `majority_class` | {max(y.mean(), 1 - y.mean()):.3f} |")

StratifiedKFold reports 0.986 — a network that has apparently solved the diagnosis. GroupKFold on the same network, the same data and the same folds count reports 0.708 against a majority-class baseline of 0.558. The 0.986 was never a diagnosis: with six of a patient's eight visits in training, the network only had to recognize the patient.

Note what would have happened in a real project. The 0.986 model ships. In production every patient is new, so the true performance is 0.708, and the gap is discovered by a clinician, not by the team.

Step 3 — a feature that peeks

An autocorrelated return series, where tomorrow's direction genuinely is partly predictable — so the honest model is legitimately better than a coin. Both rows use TimeSeriesSplit; the split is correct in both. Only the window differs.

"""Temporal leakage: a feature that looks one step into the future.

The series is an autocorrelated return, so tomorrow's direction really is
partly predictable from today's — an honest model beats the coin. The trap is
`rolling(3, center=True)`: a window centred on t spans t-1, t and t+1, so the
column quietly carries the answer. A temporal split does not save you here,
because the leak is inside the feature, not inside the split.

Printed as a markdown table, in identifiers only, so one artifact serves both
the English and the Portuguese page.
"""

import numpy as np
import pandas as pd
from sklearn.model_selection import TimeSeriesSplit, cross_val_score
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

T, LAGS, FOLDS = 1200, 8, 5

rng = np.random.default_rng(3)
r = np.zeros(T)
for t in range(1, T):                       # AR(1): today's return echoes yesterday's
    r[t] = 0.75 * r[t - 1] + 0.5 * rng.normal()

df = pd.DataFrame({"r": r})
for k in range(1, LAGS + 1):
    df[f"r_lag{k}"] = df["r"].shift(k)
df["y"] = (df["r"].shift(-1) > 0).astype(int)        # does it go up tomorrow?
df["ma3_center"] = df["r"].rolling(3, center=True).mean()   # spans t-1, t, t+1  ← leak
df["ma3_causal"] = df["r"].rolling(3).mean()               # spans t-2, t-1, t

data = df.dropna()
lags = ["r"] + [f"r_lag{k}" for k in range(1, LAGS + 1)]

model = lambda: make_pipeline(
    StandardScaler(), MLPClassifier((64, 32), max_iter=3000, random_state=0)
)
score = lambda cols: cross_val_score(
    model(), data[cols], data["y"], cv=TimeSeriesSplit(FOLDS)
).mean()

leaky = score(lags + ["ma3_center"])
honest = score(lags + ["ma3_causal"])

print(f"| `{len(data)} days, cv=TimeSeriesSplit({FOLDS})` | `acc` |")
print("|---|---:|")
print(f"| `rolling(3, center=True)` | **{leaky:.3f}** |")
print(f"| `rolling(3)` | **{honest:.3f}** |")
print(f"| `majority_class` | {max(data.y.mean(), 1 - data.y.mean()):.3f} |")

rolling(3, center=True) spans \(t-1\), \(t\) and \(t+1\): the column contains tomorrow. The network recovers it and reports 0.937 against an honest 0.716. This is the leak that survives every split-related precaution, because it was never about the split — it was one keyword argument, in a line of feature engineering, written months earlier.

Step 4 — the mild leak and the catastrophic one

The same mistake, applied to two preprocessing steps: fit on everything, then split. An imbalanced problem, 5% positives, scored with ROC AUC so that the class balance cannot distort the comparison.

"""Preprocessing before the split: the mild leak and the catastrophic one.

Two operations, the same mistake — fitted on the whole dataset instead of on
the training fold. Standardizing early moves the score by a rounding error.
Balancing the classes early moves it by a sixth of the scale, because random
oversampling literally copies minority rows, and the copies land on both sides
of the split: the model is scored on rows it memorized.

ROC AUC is the metric here precisely because it does not depend on the class
balance, so the balanced dataset and the original 5% one can be compared.

Printed as a markdown table, in identifiers only, so one artifact serves both
the English and the Portuguese page.
"""

import numpy as np
from sklearn.metrics import roc_auc_score
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler

N, DIM, POSITIVE, FOLDS = 1000, 10, 0.05, 5

rng = np.random.default_rng(5)
X = rng.normal(size=(N, DIM))
weights = np.zeros(DIM)
weights[0], weights[1] = 1.0, 0.6                    # two columns carry real signal
risk = X @ weights + 1.2 * rng.normal(size=N)
y = (risk > np.quantile(risk, 1 - POSITIVE)).astype(int)

cv = StratifiedKFold(FOLDS, shuffle=True, random_state=0)
net = lambda: MLPClassifier((32,), max_iter=2000, random_state=0)
model = lambda: make_pipeline(StandardScaler(), net())


def balance(X, y, seed=0):
    """Random oversampling: copy minority rows until the classes are even."""
    rng = np.random.default_rng(seed)
    minority, majority = np.flatnonzero(y == 1), np.flatnonzero(y == 0)
    copies = rng.choice(minority, size=len(majority) - len(minority), replace=True)
    index = np.concatenate([np.arange(len(y)), copies])
    rng.shuffle(index)
    return X[index], y[index]


# --- standardization -------------------------------------------------------
X_scaled = StandardScaler().fit_transform(X)         # WRONG: mean and std of all rows
scale_early = cross_val_score(net(), X_scaled, y, cv=cv, scoring="roc_auc").mean()
scale_late = cross_val_score(model(), X, y, cv=cv, scoring="roc_auc").mean()

# --- class balancing -------------------------------------------------------
X_balanced, y_balanced = balance(X, y)               # WRONG: copies cross the split
balance_early = cross_val_score(
    model(), X_balanced, y_balanced, cv=cv, scoring="roc_auc"
).mean()

folds = []                                           # RIGHT: copies stay in the fold
for train, test in cv.split(X, y):
    X_train, y_train = balance(X[train], y[train])
    fitted = model().fit(X_train, y_train)
    folds.append(roc_auc_score(y[test], fitted.predict_proba(X[test])[:, 1]))
balance_late = np.mean(folds)

print(f"| `n={N}, positives={POSITIVE:.0%}, cv={FOLDS}` | `auc` | `gap` |")
print("|---|---:|---:|")
print(f"| `scale_then_split` | {scale_early:.3f} | +{scale_early - scale_late:.3f} |")
print(f"| `split_then_scale` | **{scale_late:.3f}** | — |")
print(f"| `balance_then_split` | {balance_early:.3f} | +{balance_early - balance_late:.3f} |")
print(f"| `split_then_balance` | **{balance_late:.3f}** | — |")

Standardizing early costs 0.004 — a mean and a standard deviation estimated over 1 000 rows barely move when 200 of them are removed. Balancing early costs 0.173, because random oversampling copies minority rows, and the copies land on both sides of the split: the model is then scored on rows it memorized verbatim. SMOTE behaves the same way, and worse, since its synthetic points are built from neighbours that may sit in the test fold.

It happens to serious people

Leakage is not a beginner's mistake; it is a systematic one, and it has been measured.

• KDD Cup 2008 — breast cancer detection from mammograms. The patient ID turned out to be one of the most predictive features: ids had been assigned in blocks by source institution, and the institutions had very different cancer rates. The winning analysis documents the leak and the reasoning that found it.¹
• COVID-19 imaging, 2020–2021 — a systematic review of models built to detect COVID from chest radiographs and CT found that none of the models reviewed was fit for clinical use, with leakage among the recurring causes: duplicated patient images spread across train and test, and control images drawn from a different population than the positives.²
• Across science, 2023 — a survey of machine-learning-based research reports leakage affecting 294 papers across 17 fields, from medicine to political science, and proposes a model-info sheet to make the pipeline auditable.³

The common thread is not incompetence. It is that leakage produces good news, and good news is not audited with the same energy as bad news.