Machine Learning · Insper

Pipelines

One object that makes the train/test rule impossible to break by accident

Three classic bugs, measured, and the scikit-learn objects that remove them

The rule from the preprocessing lesson

  • Every step that learns from the data: the median of an imputer, the mean and standard deviation of a scaler, the categories of an encoder.
  • It is fit on the training rows only.
  • Then it is replayed, unchanged, on every other row: the test set, a validation split, next month's data in production.
Written by hand, the rule becomes a pile of fit and transform calls that must stay in the right order, on the right data, forever.

The path of this lesson

  1. The manual workflowseven calls, three bugs, measured
  2. One Pipelinefit and predict, step by step
  3. Real columnsColumnTransformer and steps of your own
  4. Inspect and trustinspection, persistence, the leak it prevents
Each stage answers one question: what goes wrong when the rule is kept by hand? → how can one object keep the rule for us? → how do mixed columns and steps of your own fit in? → what did it learn, and is the score honest?
Stage 1 · What goes wrong when the rule is kept by hand?

The data and the model

California housing

  • 20,640 districts
  • 8 numeric columns (total_bedrooms has holes)
  • 1 text column, ocean_proximity
  • target: the median house value

a black box: k-NN (k-nearest neighbors)

Predicts a district as the average of the 10 most similar training districts.

Two properties matter here: it depends on the scale of the columns, and it is fast.

Preprocessing: fill the holes with the median, then standardize. The k-NN lesson studies the model itself.

The careful manual workflow


imputer = SimpleImputer(strategy='median')
scaler = StandardScaler()
model = KNeighborsRegressor(n_neighbors=10)

A_train = scaler.fit_transform(imputer.fit_transform(X_train[num]))   # learn on train
model.fit(A_train, y_train)
A_test = scaler.transform(imputer.transform(X_test[num]))             # replay on test
pred = model.predict(A_test)

careful workflow

$43,023
mean absolute error (MAE) on 4,128 test districts

always the training mean

$92,894
the floor any real model must beat

Seven calls, and the order matters

  • fit_transform on train, transform on test.
  • Imputer before scaler.
  • The same eight columns, in the same order.

Each bug below changes one thing in the prediction code. The trained model is the same every time.

Predict. Which do you expect to be the most dangerous bug: one that crashes, one that gives a bad score, or one that gives a good score?

Bug 1 by hand: re-fitting on the data you predict

training rowsnew rows
200, NaN, 400, 600, 800NaN, 900

NaN (not a number) marks a missing value.

right: fit on train

median 500 → column 200, 500, 400, 600, 800

mean 500, std \(\sqrt{40{,}000} = 200\)

new rows: NaN → 500 → 0; 900 → \((900-500)/200\) = 2

wrong: fit again on the new rows

median 900 → both rows become 900

mean 900, no spread

new rows: 0 and 0

The district two standard deviations above the training average now looks perfectly average.

Bug 1 on the real data: it depends on how the rows arrive

the scaler is re-fit on…MAE
the whole test set at once$42,814
each region's batch$65,754
one district at a time$116,049
  • Invisible on the whole test set: 4,128 random districts have almost the training statistics; the MAE even comes out $209 lower, by chance.
  • Regional batches are re-centered on themselves: the neighbors found are no longer the right ones.
  • One row has zero spread: every district becomes the same point, one distinct prediction for 4,128 districts.

Bug 2: a step forgotten at prediction time

imputer forgotten

the model refuses: Input X contains NaN.

the lucky case: an error

scaler forgotten

$105,978

silent: raw numbers (a total_rooms of 2,000 where it learned values around 0), a confident prediction for every district

Training code and prediction code are written at different times, by different people, sometimes in different files.

Bug 3: a different column order

The model receives numbers, not names: column 5 is "population" only because it was column 5 during training. Swap households and population:

sent as an array

$101,517

a district has about three times more residents than households; the only hint is a UserWarning

sent as a DataFrame

scikit-learn compares the names with those of fit and refuses.

Keep the data in a DataFrame all the way to the model, and let the first step check the names.

Six versions of the prediction code

Test MAE of the same trained k-NN with six versions of the prediction code; three bars are worse than the always-the-mean baseline

Three of the six bars are worse than ignoring the data. The most dangerous is the second one: the bug that looks correct.

Stage 2 · How can one object keep the rule for us?

Pipeline: one estimator, many steps


pipe = make_pipeline(SimpleImputer(strategy='median'), StandardScaler(),
                     KNeighborsRegressor(n_neighbors=10))
pipe.fit(X_train[num], y_train)            # every step learns from the training rows
pred_pipe = pipe.predict(X_test[num])      # every step replays what it learned

MAE

$43,023
identical to the careful workflow

one at a time

4,128 predictions one by one = all at once

swapped columns

refused: the names must match those of fit

The three bugs, gone by construction

  • Bug 1. pipe.predict never calls fit on a transformer: the batch size stops mattering.
  • Bug 2. There is no separate prediction code in which to forget a step: the steps travel inside the object.
  • Bug 3. The pipeline hands the DataFrame to its first step, which checks the names.
The rules: every step except the last is a transformer (fit + transform); the last can be anything with fit. The pipeline exposes the methods of its last step: predict, score, predict_proba.

What fit and predict do, step by step

The toy column through a two-step pipeline: fit learns median 500, mean 500 and std 200; predict reuses them and sends 0 and 2; the fit_transform bug sends 0 and 0

fit: each transformer is fit on what it receives and transforms it for the next one; the last step is only fit. predict: only transform, with the parameters frozen at fit time.

A spy between the steps

A do-nothing step that reports, for total_bedrooms, the missing values and the mean.

callrowsmissingmean
fit · raw16,512163538.92
fit · imputed16,5120537.92
predict · raw4,12844533.66
predict · scaled4,1280−0.01
Predict. Should the scaled test column have mean exactly 0?

No: the test rows are measured with the ruler of the training rows. A test mean of exactly 0.00 would be the symptom of bug 1.

During fit, only the 16,512 training rows exist. The 44 test holes get the training median.

Pipeline or make_pipeline


pipe_named = Pipeline([('impute', SimpleImputer(strategy='median')),
                       ('scale', StandardScaler()),
                       ('model', KNeighborsRegressor(n_neighbors=10))])
  • make_pipeline names each step after its class: 'simpleimputer', 'standardscaler', 'kneighborsregressor'.
  • Pipeline takes the names from you.

Same kind of object; names only matter when you refer to a step later. Short explicit names pay off in long pipelines.

Stage 3 · How do mixed columns and steps of your own fit in?

Different columns, different treatments: ColumnTransformer


preprocess = ColumnTransformer([
    ('num', make_pipeline(SimpleImputer(strategy='median'), StandardScaler()), num),
    ('cat', OneHotEncoder(handle_unknown='ignore', sparse_output=False), ['ocean_proximity']),
])
full = make_pipeline(preprocess, KNeighborsRegressor(n_neighbors=10))
full.fit(X_train, y_train)                 # the raw 9-column table goes in

with ocean_proximity

$41,552
down from $43,023

each branch

a triple (name, transformer, columns); a branch can itself be a pipeline

Watch the columns travel

What actually comes out

Eight raw test districts in; 13 columns out: eight standardized numbers and one 1 among the five one-hot columns of each row

get_feature_names_out says what each of the 13 columns is: the prefix is the branch (num__, cat__), the one-hot columns carry the category. The missing total_bedrooms receives the training median, 437.

Which columns go where

remainder

Columns no branch mentions are dropped by default: 2 columns out.

'passthrough' appends them unchanged: 9 columns, but unscaled (and one is text).

selectors

make_column_selector(dtype_include='number') picks columns by type when fit.

They trust the types: a zip code stored as a number lands in the numeric branch.

'drop' is the safe default: a forgotten identifier never reaches the model silently. Explicit lists say exactly what happens.

Robust to column order and to new categories

  • The nine columns in reverse order: identical predictions. The ColumnTransformer selects by name, so bug 3 disappears.
  • A district relabeled 'LAKE', a category never seen: zeros in the five one-hot columns and a prediction, $90,030, the same as with its real label.
Without handle_unknown='ignore' the same row raises Found unknown categories, and one new value in production takes the whole service down. ISLAND has only 5 districts in the whole table.

Your own steps: stateless


log = FunctionTransformer(np.log1p, feature_names_out='one-to-one')   # nothing to learn
skewness of the four count columns
before3.85, 3.32, 5.17, 3.26
after \(\log(1+x)\)−1.09, −1.0, −1.04, −1.06
Its fit does nothing, so it cannot leak. 'one-to-one' keeps get_feature_names_out working.

Your own steps: a transformer that learns

A clipper: learn the 1st and 99th percentiles of each column on the training rows, pull every value outside them back to the bound.

  • __init__ only stores the hyperparameters.
  • fit learns and returns self; learned state ends with _ (lower_, upper_).
  • transform applies what was learned, after checking it was fit.
  • TransformerMixin gives fit_transform; OneToOneFeatureMixin the feature names.

learned on 16,512 districts

total_rooms: 162.2 to 11,248.0

$41,361
MAE, slightly better than $41,552

What the clipper is really for

Predict. One new district arrives with its total_rooms multiplied by 1,000 (a unit error). What happens to its prediction, with and without the clipper?
true value$65,600
correct row$90,030
typo, no clipper$251,020
typo, clipper$145,690
Predicted value of one district as total_rooms is multiplied from 1 to 1000: without the clipper it climbs to $251k; with it, it stops at $146k

Still wrong, and no transformer can fix bad input. But the damage is bounded, and the bound was learned from training data only.

Stage 4 · What did it learn, and is the score honest?

Inspection and persistence

open the container

  • full.named_steps, or indexing like a list: full[-1]
  • named_transformers_['num'][0].statistics_: the eight training medians (437 bedrooms)
  • full[:-1] = the preprocessing only: (2, 13)

one file

joblib.dump(full, 'housing-knn.joblib')

3.9 MB (k-NN keeps all the training districts); identical predictions after the round trip.

Whoever loads it cannot forget a step, re-fit a scaler or reorder the columns. Versions and safe loading are the subject of the lesson on MLOps (machine-learning operations).

The leak a pipeline prevents

Fit a step on all 20,640 districts before the split, or inside the pipeline. Two kinds of step:

a scaler

learns means and standard deviations; never looks at the price

a target encoder

replaces each zone by the average price of that zone (zones: coordinates rounded to ~1 km)

12,590 zones · 8,237 with a single district · 44% of test zones unseen in training

Predict. Which leak is bigger? Think about a test district alone in its zone.

The size of a leak depends on what the step learns

encoder fit on all rows$14,567
encoder inside the pipeline$42,552
no zone at all$60,383
Test MAE over 20 splits: for the scaler, all rows or pipeline give the same $42.3k; for the target encoder, all rows reports $14.5k against an honest $42.8k
  • A district alone in its zone: its "zone price" is its own price, and the model reads the answer off its input.
  • Over 20 splits: the scaler leak moves the MAE by $4 on average; the encoder leak makes it look $28,300 smaller.
You cannot know the size of a leak without measuring both versions. With every fitted step inside the pipeline, there is nothing to measure.

Simulator: fit on which rows?

Predict. Switch to test = later period, then to fit on all rows. Do the test rows look more or less shifted? Did the model get any better?

Looking ahead: tuning through the pipeline


full.get_params()        # 46 parameters, e.g. columntransformer__num__simpleimputer__strategy
full.set_params(kneighborsregressor__n_neighbors=20)
  • Every setting of every step is reachable as step__parameter.
  • A search that refits the whole pipeline re-learns the preprocessing on the training rows of each try.
Never choose a setting by looking at the test score: the cross-validation lesson does it honestly, with GridSearchCV and these names.

Summary

  1. Fitted steps: fit on train, replay unchanged. By hand, the rule breaks.
  2. The most dangerous bug is the one that looks correct on a notebook test set.
  3. Pipeline: two calls, the same predictions, the three bugs gone by construction.
  4. ColumnTransformer: a branch per group of columns, selected by name.
  5. Your own steps: FunctionTransformer, or a class with fit/transform.
  6. Every fitted step inside the pipeline: no leak to measure.
Design habit: write the pipeline skeleton first. The object you evaluate is the object you ship.
Lab

From raw Titanic rows to a saved model

  • Download the lab notebook and open it in Jupyter or Colab (File → Upload notebook).
  • 891 raw passengers; a k-NN classifier as the black box.
  • Run the blocks in order, in one session.

1. load and look
2. split once
3. the preprocessing, column by column

4. fit on train, score on test
5. what each step learned
6. what the model sees

7. new passengers, raw
8. save, load, compare
9. a transformer of your own: the title