Pipelines

The preprocessing lesson set a rule: every step that learns something from the data (the median of an imputer, the mean and standard deviation of a scaler, the list of categories of an encoder) is fit on the training rows only, and then replayed, unchanged, on every other row the model will ever see: the test set, a validation split, next month's data in production. Written by hand, that rule becomes a pile of fit and transform calls that must stay in the right order, on the right data, forever. This lesson measures what goes wrong when one of those calls slips, and replaces the pile with a single object, scikit-learn's Pipeline, that makes the rule impossible to break by accident.

This lesson has two approaches. The fundamental one measures what goes wrong when the rule is kept by hand and builds, step by step, the object that keeps it: a Pipeline, then a ColumnTransformer for a real table, then the leak the whole arrangement prevents. The advanced one opens that object, saves it, extends it with steps of your own and reaches every setting by name, which is what automatic tuning needs.

Every number below comes from running the code on this page; each figure is produced by a short script that sits next to it.

How this lesson is organized

The lesson has two approaches. Each has slides, text and a lab to do in class; the quiz at the end covers the fundamental one. Start with the fundamental approach: the advanced one builds on it.

Both labs use the same lab notebook: download it and open it in Jupyter or Colab (File → Upload notebook). The fundamental part runs from step 1 to 5, the advanced part from step 6 to 9.

Fundamental approach

One question drives this part: what goes wrong when the train/test rule is kept by hand, and how can one object keep it for us? The answer is measured on the California housing table: three classic bugs of a manual workflow, the Pipeline that removes them by construction, the ColumnTransformer that handles a real table with text and holes, and the size of the leak the whole arrangement prevents.

This part follows one path in four stages, drawn below: 1. the manual workflow, seven calls and three bugs, each one measured; 2. one Pipeline, which removes the three bugs by construction; 3. real columns, with ColumnTransformer routing each group of columns to its own treatment; 4. an honest score, the leak a pipeline prevents and why its size depends on what a step learns. Each stage links to its section, and in the slides a strip at the bottom shows which stage you are in.

Fundamental approach slides

Click the slides and use the ← → arrows to navigate; F opens full screen and Esc shows every slide. Open the slides in a new tab

1. The problem pipelines solve

The manual workflow

Stage 1. The data is the California housing table from the EDA lesson: 20,640 districts, eight numeric columns (one of them, total_bedrooms, with missing values), one categorical column (ocean_proximity) and the target, the median house value of the district. To keep the start simple, the first sections use only the eight numbers. The preprocessing is the one from the previous lesson: fill the holes with the median, then standardize.

import numpy as np
import pandas as pd
from sklearn.impute import SimpleImputer
from sklearn.metrics import mean_absolute_error
from sklearn.model_selection import train_test_split
from sklearn.neighbors import KNeighborsRegressor
from sklearn.preprocessing import StandardScaler

url = "https://raw.githubusercontent.com/hsandmann/biblio/refs/heads/main/ml/aula02/housing.csv"
housing = pd.read_csv(url)
X = housing.drop(columns='median_house_value')          # 9 columns: 8 numbers + ocean_proximity
y = housing['median_house_value']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)
num = ['longitude', 'latitude', 'housing_median_age', 'total_rooms',
       'total_bedrooms', 'population', 'households', 'median_income']

# the manual workflow: three objects, each fit on train, then replayed on test
imputer = SimpleImputer(strategy='median')
scaler = StandardScaler()
model = KNeighborsRegressor(n_neighbors=10)              # black box: see the note below

A_train = scaler.fit_transform(imputer.fit_transform(X_train[num]))   # learn on train
model.fit(A_train, y_train)
A_test = scaler.transform(imputer.transform(X_test[num]))             # replay on test
pred = model.predict(A_test)

def mae(p):
    return f"${mean_absolute_error(y_test, p):,.0f}"

print(f"{len(X_train)} training districts, {len(X_test)} test districts")
print("MAE, careful manual workflow:", mae(pred))
print("MAE, always the training mean:", mae(np.full(len(y_test), y_train.mean())))

16512 training districts, 4128 test districts
MAE, careful manual workflow: $43,023
MAE, always the training mean: $92,894

The score is the mean absolute error (MAE): on average, the prediction misses the true value of a test district by $43,023. A "model" that ignores the district and always answers the training mean misses by $92,894. That is the floor any real model must beat.

The careful version takes seven calls on three objects, and the order matters in both directions: fit_transform on train, transform on test, imputer before scaler, the same eight columns in the same order. Each of the three classic bugs below changes one thing in the prediction code. The trained model is the same in every case.

Bug 1 — re-fitting on the data you predict

The most common slip is one word: fit_transform where transform was meant. The step then re-learns its statistics from the rows it is supposed to transform. It is easiest to see by hand. Take one column with five training rows and two new rows:

Fit on the training rows. The median of the four known values is \((400 + 600)/2 = 500\), so the imputed column is 200, 500, 400, 600, 800. Its mean is 500 and its standard deviation is \(\sqrt{(300^2 + 0^2 + 100^2 + 100^2 + 300^2)/5} = \sqrt{40{,}000} = 200\). The new rows become: NaN → 500 → \((500 - 500)/200 = 0\), and 900 → \((900 - 500)/200 = 2\). The second new district is two standard deviations above the training average, which is information the model can use.

Re-fit on the new rows instead. Their median is 900 (the only known value), so both rows become 900; their mean is 900 and their spread is zero; both are transformed to 0. The district with 900 bedrooms, two standard deviations above the training average, now looks perfectly average. scikit-learn agrees:

toy_train = pd.DataFrame({'total_bedrooms': [200, np.nan, 400, 600, 800]})
toy_new = pd.DataFrame({'total_bedrooms': [np.nan, 900]})

imp, sc = SimpleImputer(strategy='median'), StandardScaler()
sc.fit(imp.fit_transform(toy_train))                       # learn on the 5 training rows
print("learned: median", imp.statistics_, "mean", sc.mean_, "std", sc.scale_)
print("right: transform         ", sc.transform(imp.transform(toy_new)).ravel())
print("wrong: fit_transform again", StandardScaler().fit_transform(
                                    SimpleImputer(strategy='median').fit_transform(toy_new)).ravel())

learned: median [500.] mean [500.] std [200.]
right: transform          [0. 2.]
wrong: fit_transform again [0. 0.]

(When a column has no spread at all, StandardScaler divides by 1 instead of 0, so the result is 0 rather than an error.)

The simulator walks through the same computation one step at a time: the median, the filled holes, the mean, the standard deviation and what reaches the model. Its three modes change only the rows the fit learns from: the training rows (the right way), the new rows again (this bug), or all seven rows before the split, the leak of section 4.

How much this costs on the real data depends on how the rows arrive for prediction. In a notebook, the whole test set arrives at once. In production, rows arrive in batches (one region's office sends its districts) or one at a time (a user fills in a form). The function below reproduces the bug: the trained model is the same, but the scaler is re-fit on each batch it predicts.

def predict_bug1(batch):
    """BUG 1: fit_transform where transform was meant: the scaler re-learns on the batch."""
    return model.predict(StandardScaler().fit_transform(imputer.transform(batch[num])))

print("re-fit on the whole test set:  ", mae(predict_bug1(X_test)))
by_region = pd.concat([pd.Series(predict_bug1(batch), index=batch.index)
                       for _, batch in X_test.groupby('ocean_proximity')])
print("re-fit on each region's batch: ", mae(by_region[X_test.index]))
one_by_one = [predict_bug1(X_test.iloc[[i]])[0] for i in range(len(X_test))]
print("re-fit on one district at a time:", mae(one_by_one),
      f"({len(set(one_by_one))} distinct prediction)")

re-fit on the whole test set:   $42,814
re-fit on each region's batch:  $65,754
re-fit on one district at a time: $116,049 (1 distinct prediction)

Three results, one bug:

• On the whole test set the bug is invisible. 4,128 random districts have almost the same means and standard deviations as the 16,512 training districts, so re-fitting changes almost nothing, and the MAE even comes out $209 lower than the correct one, by chance. This is why the bug survives code review: the notebook looks fine.
• On regional batches it costs $22,731 per district. Each batch is re-centered on itself. Inland districts are measured against other inland districts, so an income that is modest by the standards of the state becomes "average", and the latitude and longitude of every batch are moved to the center of the map. The neighbors the model finds are no longer the right ones.
• One district at a time, the model is worse than useless. A single row has zero spread, so every column becomes 0 (the toy table above). Every district is the same point, gets the same 10 neighbors and the same prediction: one distinct value for 4,128 districts, and an MAE of $116,049, worse than always predicting the mean.

Bug 2 — a step forgotten at prediction time

Training code and prediction code are often written at different times, by different people, sometimes in different files. A step present in one and missing in the other is the second classic:

print("scaler forgotten:", mae(model.predict(imputer.transform(X_test[num]))))
try:
    model.predict(scaler.transform(X_test[num]))           # imputer forgotten
except ValueError as err:
    print("imputer forgotten:", str(err).splitlines()[0])

scaler forgotten: $105,978
imputer forgotten: Input X contains NaN.

Forgetting the imputer is the lucky case: the missing values reach the model and it refuses them with an error. Forgetting the scaler is silent. The model receives raw numbers (a total_rooms of 2,000 where it learned values of a few units around 0), computes distances dominated by the largest columns, and returns a confident prediction for every district. The MAE, $105,978, is again worse than predicting the mean, and nothing on the screen says so.

Bug 3 — a different column order

The model receives numbers, not names: column 5 of the array is "population" only because it was column 5 during training. If the data arrives in another order (a new export from the database, a different API client), each column gets the statistics and the neighbors of another. Here two count columns trade places:

swapped = ['longitude', 'latitude', 'housing_median_age', 'total_rooms',
           'total_bedrooms', 'households', 'population', 'median_income']   # last-but-two swapped
X_new = X_test[swapped]
print("swapped, sent as an array:", mae(model.predict(scaler.transform(imputer.transform(X_new.to_numpy())))))
try:
    imputer.transform(X_new)                                # sent as a DataFrame
except ValueError as err:
    print("swapped, sent as a DataFrame:", " ".join(str(err).splitlines()[:2]))

swapped, sent as an array: $101,517
swapped, sent as a DataFrame: The feature names should match those that were passed during fit. Feature names must be in the same order as they were in fit.

As an array, the swap costs $101,517 per district. A district has about three times more residents than households, so the "households" column is suddenly three times larger than anything the model saw and the "population" column three times smaller. The only hint is a UserWarning ("X does not have valid feature names") among the usual notebook warnings. As a DataFrame, scikit-learn compares the column names with the ones it saw in fit and refuses. Keep the data in a DataFrame all the way to the model, and let the first step check the names.

The figure puts the six results side by side (figure script):

Three of the six bars are worse than the baseline that ignores the data, and the most dangerous bar is the second one, the bug that looks correct.

2. Pipeline: one estimator, many steps

Stage 2. A Pipeline is a list of steps that behaves as a single estimator. make_pipeline builds one from the steps and names them automatically:

from sklearn.pipeline import make_pipeline

pipe = make_pipeline(SimpleImputer(strategy='median'), StandardScaler(),
                     KNeighborsRegressor(n_neighbors=10))
pipe.fit(X_train[num], y_train)            # every step learns from the training rows
pred_pipe = pipe.predict(X_test[num])      # every step replays what it learned
print("MAE:", mae(pred_pipe), "· identical to the careful manual workflow:",
      np.array_equal(pred_pipe, pred))

one_by_one = [pipe.predict(X_test[num].iloc[[i]])[0] for i in range(len(X_test))]
print("one district at a time == all at once:", np.allclose(one_by_one, pred_pipe))
try:
    pipe.predict(X_new)                    # the swapped columns of bug 3
except ValueError as err:
    print("swapped columns:", str(err).splitlines()[0])
print(list(pipe.named_steps))

MAE: $43,023 · identical to the careful manual workflow: True
one district at a time == all at once: True
swapped columns: The feature names should match those that were passed during fit.
['simpleimputer', 'standardscaler', 'kneighborsregressor']

Two calls, fit and predict, replace seven, the predictions are exactly those of the careful manual workflow, and the three bugs are gone by construction:

• Bug 1. pipe.predict never calls fit on a transformer, so the batch size stops mattering: 4,128 predictions made one district at a time are identical to the 4,128 made at once.
• Bug 2. There is no separate prediction code in which to forget a step. The steps travel inside the object.
• Bug 3. The pipeline hands the DataFrame to its first step, which checks the column names and refuses the swapped table.

The rules of composition are few. Every step except the last must be a transformer (it has fit and transform); the last step can be anything with fit, usually a model with predict. The pipeline then exposes the methods of its last step: pipe.predict, pipe.score, and pipe.predict_proba for a classifier.

What fit and predict do, step by step

pipe.fit(X, y) walks down the list. Each transformer is fit on what it receives and then transforms it (fit_transform); its output is the input of the next step; the last step is only fit. pipe.predict(X) walks the same list calling only transform, with the parameters frozen at fit time, and ends with the model's predict. The toy column of bug 1, drawn as a pipeline (figure script):

To watch it happen on the real data, put a do-nothing step that reports what it sees between the real steps (how to write such a class is the subject of your own steps, in the advanced approach). It prints the shape of the data and, for the total_bedrooms column, the number of missing values and the mean at that point:

from sklearn.base import BaseEstimator, TransformerMixin

class Spy(BaseEstimator, TransformerMixin):
    """A step that changes nothing and reports what passes through it."""
    def __init__(self, label=''):
        self.label = label

    def fit(self, X, y=None):
        self._report('fit', X)
        return self

    def transform(self, X):
        self._report('transform', X)
        return X

    def _report(self, call, X):
        X = np.asarray(X, dtype=float)
        col = X[:, 4]                                            # total_bedrooms
        print(f"  {self.label:8s} {call:9s} {str(X.shape):11s} "
              f"missing {np.isnan(col).sum():3d} · mean {np.nanmean(col):7.2f}")

traced = make_pipeline(Spy('raw'), SimpleImputer(strategy='median'), Spy('imputed'),
                       StandardScaler(), Spy('scaled'), KNeighborsRegressor(n_neighbors=10))
print("traced.fit(X_train, y_train)")
traced.fit(X_train[num], y_train)
print("traced.predict(X_test)")
_ = traced.predict(X_test[num])

traced.fit(X_train, y_train)
  raw      fit       (16512, 8)  missing 163 · mean  538.92
  raw      transform (16512, 8)  missing 163 · mean  538.92
  imputed  fit       (16512, 8)  missing   0 · mean  537.92
  imputed  transform (16512, 8)  missing   0 · mean  537.92
  scaled   fit       (16512, 8)  missing   0 · mean    0.00
  scaled   transform (16512, 8)  missing   0 · mean    0.00
traced.predict(X_test)
  raw      transform (4128, 8)   missing  44 · mean  533.66
  imputed  transform (4128, 8)   missing   0 · mean  532.63
  scaled   transform (4128, 8)   missing   0 · mean   -0.01

Read it from top to bottom:

• During fit, only training rows exist. Every step sees 16,512 rows, never 4,128. Each step is fit and then immediately transforms (the fit + transform pairs), because the next step must be fit on transformed training data: the scaler learns its mean from imputed values (537.92), not from raw ones with holes.
• The imputer fills 163 holes with the training median, which pulls the mean of the column down by one bedroom, from 538.92 to 537.92: the median (437) is below the mean because the column has a long right tail.
• During predict, nobody calls fit. The 44 holes of the test set are filled with the training median, and the scaler subtracts the training mean. That is why the scaled test column has mean −0.01 and not exactly 0: the test rows are measured with the ruler of the training rows, which is the honest thing to do. A mean of exactly 0.00 on the test set would be the symptom of bug 1.

Pipeline or make_pipeline

make_pipeline names each step after its class, in lower case: 'simpleimputer', 'standardscaler', 'kneighborsregressor' (and 'spy-1', 'spy-2', ... when a class repeats). Pipeline takes the names from you, as a list of (name, step) pairs:

from sklearn.pipeline import Pipeline

pipe_named = Pipeline([('impute', SimpleImputer(strategy='median')),
                       ('scale', StandardScaler()),
                       ('model', KNeighborsRegressor(n_neighbors=10))])

Both build the same kind of object; the names only matter when you refer to a step later (to inspect it, or to change one of its settings). Short explicit names pay off in long pipelines.

3. Heterogeneous columns: ColumnTransformer

Stage 3. A pipeline applies every step to every column. Real tables need different treatments for different columns: here the eight numbers need the median and the scaler, and ocean_proximity, a text column, needs one-hot encoding. A ColumnTransformer routes each group of columns to its own branch and puts the results side by side:

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder

preprocess = ColumnTransformer([
    ('num', make_pipeline(SimpleImputer(strategy='median'), StandardScaler()), num),
    ('cat', OneHotEncoder(handle_unknown='ignore', sparse_output=False), ['ocean_proximity']),
])
full = make_pipeline(preprocess, KNeighborsRegressor(n_neighbors=10))
full.fit(X_train, y_train)                 # the raw 9-column table goes in
print("MAE with ocean_proximity:", mae(full.predict(X_test)))
print(preprocess.get_feature_names_out())

MAE with ocean_proximity: $41,552
['num__longitude' 'num__latitude' 'num__housing_median_age'
 'num__total_rooms' 'num__total_bedrooms' 'num__population'
 'num__households' 'num__median_income' 'cat__ocean_proximity_<1H OCEAN'
 'cat__ocean_proximity_INLAND' 'cat__ocean_proximity_ISLAND'
 'cat__ocean_proximity_NEAR BAY' 'cat__ocean_proximity_NEAR OCEAN']

Each branch is a triple (name, transformer, columns), and a branch can itself be a pipeline, as 'num' is. The whole thing is one transformer, so it goes into a pipeline like any other step, and the model now receives the raw 9-column table: the text column is no longer a problem, and using it lowers the MAE from $43,023 to $41,552.

get_feature_names_out says what each output column is. The prefix is the name of the branch that produced it (num__, cat__), and the one-hot columns carry the category (cat__ocean_proximity_INLAND). Without it, the model's input is a 13-column block of anonymous numbers. What actually comes out, for eight test districts (figure script):

Which columns go where: lists, remainder, selectors

Columns that no branch mentions are dropped by default (remainder='drop'). With remainder='passthrough' they are appended unchanged:

two = ColumnTransformer([('num', StandardScaler(), ['median_income', 'housing_median_age'])])
print("remainder='drop':       ", two.fit_transform(X_train).shape)
two_plus = ColumnTransformer([('num', StandardScaler(), ['median_income', 'housing_median_age'])],
                             remainder='passthrough')
print("remainder='passthrough':", two_plus.fit_transform(X_train).shape)
print(two_plus.get_feature_names_out()[:4])

remainder='drop':        (16512, 2)
remainder='passthrough': (16512, 9)
['num__median_income' 'num__housing_median_age' 'remainder__longitude'
 'remainder__latitude']

'drop' is the safe default: an identifier column or a free-text field you forgot about never reaches the model silently. 'passthrough' is for columns that are already ready. Here it would be a mistake, because the seven passed-through columns arrive unscaled (and ocean_proximity arrives as text).

Instead of lists of names, a branch can take a selector that chooses columns by type when the transformer is fit:

from sklearn.compose import make_column_selector

by_type = ColumnTransformer([
    ('num', make_pipeline(SimpleImputer(strategy='median'), StandardScaler()),
     make_column_selector(dtype_include='number')),
    ('cat', OneHotEncoder(handle_unknown='ignore', sparse_output=False),
     make_column_selector(dtype_exclude='number')),
])
by_type.fit(X_train)
for name, _, columns in by_type.transformers_:
    print(name, columns)

num ['longitude', 'latitude', 'housing_median_age', 'total_rooms', 'total_bedrooms', 'population', 'households', 'median_income']
cat ['ocean_proximity']

The fitted transformer (transformers_, with the trailing underscore) records the columns each selector picked. Selectors are convenient for wide tables, but they trust the types: a zip code stored as a number lands in the numeric branch. Explicit lists are more work and say exactly what happens.

Seeing the output: feature names and set_output

By default a transformer returns a NumPy array. set_output(transform='pandas') makes it return a DataFrame with the names of get_feature_names_out as columns, which is the easiest way to check a preprocessing by eye:

preprocess.set_output(transform='pandas')
print(preprocess.transform(X_test.head(3)).round(2).T)

                                 20046  3024   15663
num__longitude                   -1.41   0.60   1.16
num__latitude                     2.36  -0.84  -1.36
num__housing_median_age          -1.47   0.51   0.82
num__total_rooms                 -0.17  -0.67   0.21
num__total_bedrooms              -0.23  -0.80  -0.26
num__population                  -0.14  -0.73  -0.22
num__households                  -0.16  -0.76  -0.18
num__median_income               -0.87   0.45   1.96
cat__ocean_proximity_<1H OCEAN    0.00   1.00   0.00
cat__ocean_proximity_INLAND       1.00   0.00   0.00
cat__ocean_proximity_ISLAND       0.00   0.00   0.00
cat__ocean_proximity_NEAR BAY     0.00   0.00   0.00
cat__ocean_proximity_NEAR OCEAN   0.00   0.00   1.00

Three test districts, transposed so that each column of the output is a row here. District 20046 is inland, 2.36 standard deviations north of the average training district, with an income 0.87 standard deviations below average. The index of the DataFrame is preserved, so every transformed row can be traced back to its raw row. (pandas output needs dense arrays, hence sparse_output=False in the encoder.)

Robust to column order and to new categories

A ColumnTransformer fit on a DataFrame selects its columns by name, so bug 3 goes away completely instead of turning into an error. And an encoder with handle_unknown='ignore' turns a category it never saw into a row of zeros instead of crashing:

reversed_cols = X_test[X_test.columns[::-1]]              # same columns, opposite order
print("reversed columns, same predictions:", np.array_equal(full.predict(reversed_cols), full.predict(X_test)))
new = X_test.head(1).assign(ocean_proximity='LAKE')        # a category never seen in training
print(preprocess.transform(new).filter(like='ocean').to_numpy(), full.predict(new).round())

reversed columns, same predictions: True
[[0. 0. 0. 0. 0.]] [90030.]

With all nine columns in reverse order, the predictions are identical. The district relabeled 'LAKE' gets zeros in the five ocean_proximity columns and a prediction, $90,030, the same as with its real label, INLAND: its 10 nearest neighbors are so close in the eight numbers that losing one category column did not change them. Without handle_unknown='ignore', the same row would raise ValueError: Found unknown categories, and one new value in production would take the whole service down. Rare categories make this likely even without typos: ISLAND has 5 districts in the whole table, all of which landed in the training set with this split.

4. The leak a pipeline prevents

Stage 4. The preprocessing lesson stated that fitting a step on all the data before the split leaks information from the test set. How much depends on what the step learns. Compare two steps.

The first is the scaler. Twenty different splits, each done twice: once with the imputer and the scaler fit on all 20,640 districts before the split, once inside the pipeline. The second is a step that learns from the target: a target encoder, which replaces each category of a column by the average target of that category in the training data. It is the standard tool for a categorical column with thousands of values, where one-hot encoding would create thousands of columns. The table has no such column, so we build one from the coordinates, rounded to two decimals (cells of about 1 km, playing the part of a zip code):

from sklearn.preprocessing import TargetEncoder

zone = housing['latitude'].round(2).astype(str) + '_' + housing['longitude'].round(2).astype(str)
Z = X[['median_income', 'housing_median_age']].assign(zone=zone)   # ~1 km cells, like a zip code
Z_train, Z_test = Z.loc[X_train.index], Z.loc[X_test.index]
print(zone.nunique(), "zones ·", (zone.value_counts() == 1).sum(), "with a single district ·",
      f"{(~Z_test['zone'].isin(Z_train['zone'])).mean():.0%} of test zones unseen in training")

# LEAK: the encoder learns every zone's mean price from ALL rows, test prices included
enc = TargetEncoder(target_type='continuous', random_state=0).fit(Z[['zone']], y)
def encode(D):
    return D.drop(columns='zone').assign(zone_price=enc.transform(D[['zone']])[:, 0])
leaky = make_pipeline(StandardScaler(), KNeighborsRegressor(n_neighbors=10))
leaky.fit(encode(Z_train), y_train)
print("encoder fit on all rows:    ", mae(leaky.predict(encode(Z_test))))

# HONEST: the encoder is a pipeline step, so it only ever sees training prices
honest = make_pipeline(
    ColumnTransformer([('zone', TargetEncoder(target_type='continuous', random_state=0), ['zone'])],
                      remainder='passthrough'),
    StandardScaler(), KNeighborsRegressor(n_neighbors=10))
honest.fit(Z_train, y_train)
print("encoder inside the pipeline:", mae(honest.predict(Z_test)))
no_zone = make_pipeline(StandardScaler(), KNeighborsRegressor(n_neighbors=10))
no_zone.fit(Z_train.drop(columns='zone'), y_train)
print("no zone at all:             ", mae(no_zone.predict(Z_test.drop(columns='zone'))))

12590 zones · 8237 with a single district · 44% of test zones unseen in training
encoder fit on all rows:     $14,567
encoder inside the pipeline: $42,552
no zone at all:              $60,383

The leaky version reports an MAE of $14,567, almost three times smaller than any other error in this lesson, from two columns and a zone code. The mechanism is simple. 8,237 of the 12,590 zones contain a single district. When the encoder is fit on all the rows, the "average price of the zone" of such a test district is its own price (pulled a little toward the global average by the encoder's smoothing), and the model reads the answer off its input. Inside the pipeline the encoder sees only training prices: 44% of the test districts fall in zones it never saw and get the global average, the others get the average of other districts in their zone. The honest MAE is $42,552. The zone is still useful (without it, $60,383), just not magical.

(One detail of this encoder: in fit_transform, which is what a pipeline calls during fit, TargetEncoder encodes each training row with averages computed without that row, using an internal split. This keeps the model from learning to trust the encoding too much. fit followed by transform, as in the leaky version, does not do it.)

Repeated over 20 splits, the contrast between the two kinds of step is stable (figure script):

For the scaler, the leak changes the MAE by $4 on average, and by at most $96 in any split: the mean and standard deviation of 16,512 or of 20,640 districts are practically the same numbers, and the scaler never looks at the price. For the target encoder, the leak makes the error look $28,300 smaller on average, and at least $27,100 smaller in every split. The rule that follows is not "scalers are safe". It is: you cannot know the size of a leak without measuring both versions, and with every fitted step inside the pipeline there is nothing to measure. The more a step uses the target (target encoding, feature selection, oversampling), the bigger the potential leak; the cross-validation and data leakage lesson measures more of them.

The toy column of bug 1 shows the mechanism in miniature. Fit the same two steps on all seven rows, the five training rows and the two new ones, as a leaky workflow would before the split:

toy_all = pd.concat([toy_train, toy_new])                  # the 5 training rows + the 2 new ones
leaky_toy = make_pipeline(SimpleImputer(strategy='median'), StandardScaler()).fit(toy_all)
print("learned on all 7 rows: median", leaky_toy[0].statistics_,
      "mean", leaky_toy[1].mean_.round(2), "std", leaky_toy[1].scale_.round(2))
print("new rows:", leaky_toy.transform(toy_new).ravel().round(2), "instead of [0. 2.]")

learned on all 7 rows: median [600.] mean [585.71] std [216.65]
new rows: [0.07 1.45] instead of [0. 2.]

The median moves from 500 to 600, the mean from 500 to 585.71 and the standard deviation from 200 to 216.65, because 900, above every training value, is now part of the ruler that measures it. The 900-bedroom district comes out 1.45 standard deviations above the mean instead of 2: it looks less unusual than it is. The third mode of the step-by-step simulator walks through these numbers. Here the two new rows are 2 of 7, and one of them is extreme, so the leak shows; 4,128 random districts among 20,640 look like the others, and the scaler leak shrinks to the $4 measured above.

One column is enough to see why the scaler barely leaked on a random split and when it would stop being harmless. The simulator fits the lesson's two steps (median imputation, then the scaler) on a column with holes, in one of two ways: on the training rows only (as the pipeline does) or on all rows before the split. Compare the two modes on a random split; then switch the test set to a later period in which prices rose.

On a random split the test rows look like the training rows, so adding them to the fit barely changes the statistics: that is the small scaler leak measured above. When the test set comes from a later period, the leaked mean and standard deviation already contain the new prices: the shift looks smaller than it is, and the filled holes land a little closer to the truth (a median resists a minority of high values). Nothing about the model improved; only the evaluation did. Inside the pipeline, the test rows never take part in a fit, whatever the split.

5. Summary: the rule, kept by an object

1. Every step that learns (the median of an imputer, the mean and standard deviation of a scaler, the categories of an encoder) is fit on the training rows only and replayed, unchanged, on every other row.
2. By hand, the rule breaks in three classic ways: re-fitting on the rows being predicted (invisible on a large test set, $116,049 one district at a time), a step forgotten at prediction time (silent, $105,978), and another column order ($101,517 when sent as an array).
3. A Pipeline fits each step on the output of the previous one with fit_transform and, at prediction time, only calls transform. Same predictions as the careful manual code, and the three bugs are gone by construction.
4. A ColumnTransformer gives each group of columns its own branch, selects them by name and drops what no branch mentions; handle_unknown='ignore' turns a new category into zeros instead of an error.
5. The size of a leak depends on what the step learns: $4 for a scaler, $28,300 for a target encoder, on the same data. With every fitted step inside the pipeline there is nothing to measure.

Fundamental lab — from raw Titanic rows to an honest score

The preprocessing handout ended with a ColumnTransformer that turns the 891 raw Titanic passengers into a matrix of numbers, without training any model. This lab finishes the job: the same kind of preprocessing, now in a pipeline with a model at the end, fit on training rows, evaluated once on the test rows and opened up to check where each learned value came from. Opening, saving and extending it are left to the advanced lab. The model is again a black box, a k-NN classifier with its default settings (it predicts "survived" when most of the 5 most similar training passengers survived; see k-NN).

Step 1 — load and look

Predict. Which columns of a raw passenger list do you expect to have holes? Which ones could a model not use as they are?

import numpy as np
import pandas as pd

url = "https://raw.githubusercontent.com/hsandmann/biblio/refs/heads/main/ml/aula04/titanic/train.csv"
df = pd.read_csv(url)
print(df.shape)
print("missing:", df.isna().sum()[lambda s: s > 0].to_dict())
print(df[['Name', 'Sex', 'Age', 'Fare', 'Cabin', 'Embarked']].head(3).to_string())

(891, 12)
missing: {'Age': 177, 'Cabin': 687, 'Embarked': 2}
                                                  Name     Sex   Age     Fare Cabin Embarked
0                              Braund, Mr. Owen Harris    male  22.0   7.2500   NaN        S
1  Cumings, Mrs. John Bradley (Florence Briggs Thayer)  female  38.0  71.2833   C85        C
2                               Heikkinen, Miss. Laina  female  26.0   7.9250   NaN        S

891 passengers and 12 columns: identifiers (PassengerId, Name, Ticket), numbers (Age, SibSp, Parch, Fare), categories stored as text or as numbers (Sex, Embarked, Pclass), the cabin and the target Survived. Three columns have holes: Age in 177 rows (20%), Cabin in 687 (77%) and Embarked in 2. The first three rows already show everything the preprocessing must handle: text, NaN, and fares on a skewed scale (7.25 against 71.28). Nothing is cleaned by hand. The pipeline receives this table as it is, now and in production.

Step 2 — split once

Predict. With stratify=y, how close will the survival rates of the training and test rows be? And why keep in X the columns the model will never use, such as Name and Ticket?

from sklearn.model_selection import train_test_split

X = df.drop(columns='Survived')              # all 11 raw columns, text and holes included
y = df['Survived']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, stratify=y, random_state=42)
print(X_train.shape, X_test.shape)
print(f"survival rate: train {y_train.mean():.3f} · test {y_test.mean():.3f}")

(712, 11) (179, 11)
survival rate: train 0.383 · test 0.385

712 passengers to train, 179 to test, both with a survival rate of about 38% thanks to stratify. X keeps all 11 raw columns, including those the model will not use: deciding what to use is the pipeline's job, so the table given to it at prediction time can be the same raw table. The test rows are not touched until the score of step 4.

Step 3 — the preprocessing, column by column

Predict. Which of the 11 raw columns will reach the model, and what happens to the ones no branch mentions? Has anything been fit when this cell ends?

from sklearn.compose import ColumnTransformer
from sklearn.impute import SimpleImputer
from sklearn.neighbors import KNeighborsClassifier
from sklearn.pipeline import Pipeline, make_pipeline
from sklearn.preprocessing import FunctionTransformer, OneHotEncoder, StandardScaler

preprocess = ColumnTransformer([
    ('num',  make_pipeline(SimpleImputer(strategy='median'), StandardScaler()),
             ['Age', 'SibSp', 'Parch']),
    ('fare', make_pipeline(SimpleImputer(strategy='median'),
                           FunctionTransformer(np.log1p, feature_names_out='one-to-one'),
                           StandardScaler()),
             ['Fare']),
    ('cat',  make_pipeline(SimpleImputer(strategy='most_frequent'),
                           OneHotEncoder(handle_unknown='ignore', sparse_output=False)),
             ['Pclass', 'Sex', 'Embarked']),
])
model = Pipeline([('prep', preprocess), ('knn', KNeighborsClassifier())])

used = [c for _, _, cols in preprocess.transformers for c in cols]
for name, _, cols in preprocess.transformers:
    print(f"{name:4s} <- {cols}")
print("dropped by remainder='drop':", [c for c in X.columns if c not in used])

num  <- ['Age', 'SibSp', 'Parch']
fare <- ['Fare']
cat  <- ['Pclass', 'Sex', 'Embarked']
dropped by remainder='drop': ['PassengerId', 'Name', 'Ticket', 'Cabin']

Three branches, as in the handout: numbers get the median and the scaler; the fare, which is very skewed, gets the median, then \(\log(1 + x)\), then the scaler; the categories get the most frequent value and one-hot encoding (Pclass is treated as a category, with no assumption that 2nd class is "halfway" between 1st and 3rd). The loop prints the routing. The four columns that no branch mentions are dropped by remainder='drop': the three identifiers and Cabin. They are in X, and they will never reach the model. Nothing has been fit yet: model is a recipe. In Jupyter, a cell containing just model draws it as a diagram.

Step 4 — fit on train, score on test

Predict. How many separate fit calls does the single model.fit trigger? Will a k-NN with its default settings beat the "always did not survive" baseline by a lot or by a little?

from sklearn.dummy import DummyClassifier

model.fit(X_train, y_train)
baseline = DummyClassifier(strategy='most_frequent').fit(X_train, y_train)
print(f"test accuracy · pipeline {model.score(X_test, y_test):.3f} · "
      f"always 'did not survive' {baseline.score(X_test, y_test):.3f}")

test accuracy · pipeline 0.782 · always 'did not survive' 0.615

One call to fit fits three imputers, two scalers, the logarithm step, the encoder and the classifier, in the right order, all on the 712 training passengers. On the 179 test passengers the pipeline is right 78.2% of the time, against 61.5% for a "model" that always answers the majority class. There was nothing to tune (the classifier keeps its default settings), so the test set is used once, here.

Simulation: the same recipe without the scalers. Build a second ColumnTransformer in which the 'num' and 'fare' branches keep only their imputer (no logarithm, no scaler), put it in a pipeline with the same KNeighborsClassifier(), fit it on the training rows and compare its test accuracy with the 0.782 of the pipeline. Which column dominates the distances when nothing is scaled?

bare = ColumnTransformer([
    ('num',  SimpleImputer(strategy='median'), ['Age', 'SibSp', 'Parch']),     # no scaler
    ('fare', SimpleImputer(strategy='median'), ['Fare']),                       # no log, no scaler
    ('cat',  make_pipeline(SimpleImputer(strategy='most_frequent'),
                           OneHotEncoder(handle_unknown='ignore', sparse_output=False)),
             ['Pclass', 'Sex', 'Embarked']),
])
unscaled = Pipeline([('prep', bare), ('knn', KNeighborsClassifier())]).fit(X_train, y_train)
print(f"test accuracy · with scalers {model.score(X_test, y_test):.3f} · without {unscaled.score(X_test, y_test):.3f}")

test accuracy · with scalers 0.782 · without 0.676

Step 5 — what each step learned

Predict. The median age of all 891 passengers is 28.0. Has the imputer inside the pipeline learned 28.0, or another number? And how many rows has the scaler seen?

prep = model.named_steps['prep']
num_imputer, num_scaler = prep.named_transformers_['num']
print("Age, SibSp, Parch medians:", num_imputer.statistics_)
print("means after imputing:     ", num_scaler.mean_.round(3))
print("standard deviations:      ", num_scaler.scale_.round(3))
print("rows the scaler saw:      ", num_scaler.n_samples_seen_)
print("Age median, train only vs all 891 rows:", X_train['Age'].median(), "vs", X['Age'].median())
fare_imputer, _, fare_scaler = prep.named_transformers_['fare']
print(f"Fare: median {fare_imputer.statistics_[0]:.2f} · mean of log1p(Fare) {fare_scaler.mean_[0]:.3f}")
print("categories:", [c.tolist() for c in prep.named_transformers_['cat'][-1].categories_])

Age, SibSp, Parch medians: [28.5  0.   0. ]
means after imputing:      [29.556  0.493  0.39 ]
standard deviations:       [13.016  1.06   0.838]
rows the scaler saw:       712.0
Age median, train only vs all 891 rows: 28.5 vs 28.0
Fare: median 14.45 · mean of log1p(Fare) 2.951
categories: [[1, 2, 3], ['female', 'male'], ['C', 'Q', 'S']]

Every learned value can be read, and each one is proof of where it was learned:

• The age median is 28.5, not 28.0. 28.0 is the median of all 891 passengers; 28.5 is the median of the 712 training passengers. The imputer never saw the test rows. n_samples_seen_ confirms it for the scaler: 712.
• The scaler learned from imputed values. The mean age, 29.556, is the mean of the training ages after the holes were filled with 28.5, because the pipeline fits each step on the output of the previous one.
• The fare branch learned in its own units. Its imputer learned a median fare of £14.45, and its scaler learned the mean of \(\log(1 + \text{fare})\), 2.951, because the scaler sits after the logarithm.
• The encoder learned three lists of categories. Any other value at prediction time is "unknown" (step 7).

Simulation: the preprocessing fit on all 891 passengers. Reproduce the leak of section 4 on the Titanic: fit a copy of preprocess on all of X (use clone from sklearn.base), fit a KNeighborsClassifier() on the transformed training rows only, and score it on the transformed test rows. What median age does the copy learn? Does the test accuracy change? Then fit a copy of the whole model on all 891 rows and score it on the test rows.

from sklearn.base import clone

prep_all = clone(preprocess).fit(X)                     # LEAK: the preprocessing sees all 891 rows
knn_all = KNeighborsClassifier().fit(prep_all.transform(X_train), y_train)
print("Age median learned on all rows:", prep_all.named_transformers_['num'][0].statistics_[0])
print(f"test accuracy · honest {model.score(X_test, y_test):.3f} · "
      f"preprocessing fit on all rows {knn_all.score(prep_all.transform(X_test), y_test):.3f}")
whole = clone(model).fit(X, y)                          # far worse: the classifier sees the test rows
print(f"whole pipeline fit on all rows: {whole.score(X_test, y_test):.3f}")

Age median learned on all rows: 28.0
test accuracy · honest 0.782 · preprocessing fit on all rows 0.782
whole pipeline fit on all rows: 0.860

To close the fundamental part. In one sentence each: (1) why is the bug that re-fits a scaler on the test set the most dangerous of the three? (2) What does model.predict do to each transformer, and why does that make the size of the batch irrelevant? (3) Why did fitting the scaler on all rows barely change the score, while the target encoder made it look about three times better?

Your turn (fundamental)

At home:

1. By hand. A column has training values 10, NaN, 30, 50 and new values NaN, 70. Compute the median, mean and standard deviation that SimpleImputer(strategy='median') followed by StandardScaler() learn, and the two transformed new values. Check with a pipeline. Then compute what fit_transform on the two new rows would give.
2. Bug 1 with another scaler. Repeat the regional-batch experiment of bug 1 with MinMaxScaler instead of StandardScaler, in both the model and the bug. Is the damage larger or smaller? Which statistic of a batch does min–max re-learn, and why is it more fragile than a mean?
3. The size of a leak. Repeat the target-encoding experiment of section 4 with zones rounded to one decimal (about 10 km) instead of two. How many zones have a single district now? Does the gap between the leaky and the honest MAE grow or shrink, and why?
4. Break it on purpose. Predict on X_test with its columns in reverse order; on X_test.to_numpy(); and on X_test.drop(columns='Cabin'). Which of the three work, which fail, and what in the pipeline explains each result?

Advanced approach

The fundamental part built one honest object and measured what it prevents. This part opens it: it reads what each step learned and saves the whole chain as one file, adds steps of your own, with nothing to learn or with a state learned on the training rows, and reaches every setting of every step by name, which is what a search needs. The code blocks continue those of sections 1 to 4.

This part follows one path in three stages, drawn below: 1. open and save the fitted pipeline; 2. steps of your own, stateless and stateful; 3. tune through it, with the step__parameter names. Each stage links to its section, and in the slides a strip at the bottom shows which stage you are in.

Advanced approach slides

Click the slides and use the ← → arrows to navigate; F opens full screen and Esc shows every slide. Open the slides in a new tab

6. Inspection and persistence

Stage 1. A fitted pipeline is a container you can open. Steps are reached by name (named_steps) or by position, with the same indexing as a Python list, and a slice is a smaller pipeline:

print(list(full.named_steps))
print(full[-1])                                           # indexing, like a list
print(full.named_steps['columntransformer'].named_transformers_['num'][0].statistics_.round(1))
print(full[:-1].transform(X_test.head(2)).shape)          # everything but the model

['columntransformer', 'kneighborsregressor']
KNeighborsRegressor(n_neighbors=10)
[-118.5   34.2   29.  2129.   437.  1169.   411.     3.5]
(2, 13)

named_transformers_ does the same inside a ColumnTransformer, so any learned value can be read: here the eight training medians the imputer will use to fill holes (437 bedrooms, as in the figure). full[:-1] is "the preprocessing only", useful to look at exactly what the model receives.

In Jupyter, a pipeline left as the last expression of a cell is drawn as an interactive diagram (set_config(display='diagram'), the default), with each box expandable to its parameters. print gives the text version:

from sklearn import set_config

set_config(display='diagram')                             # the default: a clickable diagram in Jupyter
print(full)

Pipeline(steps=[('columntransformer',
                 ColumnTransformer(transformers=[('num',
                                                  Pipeline(steps=[('simpleimputer',
                                                                   SimpleImputer(strategy='median')),
                                                                  ('standardscaler',
                                                                   StandardScaler())]),
                                                  ['longitude', 'latitude',
                                                   'housing_median_age',
                                                   'total_rooms',
                                                   'total_bedrooms',
                                                   'population', 'households',
                                                   'median_income']),
                                                 ('cat',
                                                  OneHotEncoder(handle_unknown='ignore',
                                                                sparse_output=False),
                                                  ['ocean_proximity'])])),
                ('kneighborsregressor', KNeighborsRegressor(n_neighbors=10))])

Only the settings that differ from the defaults are shown.

Because the whole chain is one object, it is saved as one file, and the file contains the learned state of every step:

import os
import joblib

joblib.dump(full, 'housing-knn.joblib')                   # one file: preprocessing + model
loaded = joblib.load('housing-knn.joblib')
print(f"{os.path.getsize('housing-knn.joblib') / 1e6:.1f} MB on disk")
print("identical predictions after the round trip:",
      np.array_equal(loaded.predict(X_test), full.predict(X_test)))

3.9 MB on disk
identical predictions after the round trip: True

The loaded pipeline takes raw rows and returns the same predictions, to the last digit. Whoever uses it cannot forget a step, re-fit a scaler or reorder the columns: the three bugs of the first section need separate pieces of code to happen, and there are none. (The file is large because k-NN keeps all 16,512 training districts to find neighbors; most models store far less.) A saved pipeline also brings its own risks, such as library versions and the safety of loading someone else's file, and those are the subject of MLOps.

7. Your own steps

Stage 2. The fundamental part used only the steps scikit-learn ships. Anything that learns from data and transforms it can become a pipeline step. There are two ways, depending on whether the step has something to learn.

Stateless: FunctionTransformer

The preprocessing lesson recommended \(\log(1 + x)\) for right-skewed columns. A logarithm learns nothing from the data, so FunctionTransformer turns the function into a step directly:

from sklearn.preprocessing import FunctionTransformer

counts = ['total_rooms', 'total_bedrooms', 'population', 'households']
log = FunctionTransformer(np.log1p, feature_names_out='one-to-one')   # nothing to learn
print("skewness before:", X_train[counts].skew().round(2).to_list())
print("skewness after: ", log.fit_transform(X_train[counts]).skew().round(2).to_list())

skewness before: [3.85, 3.32, 5.17, 3.26]
skewness after:  [-1.09, -1.0, -1.04, -1.06]

The four count columns go from a strong right skew (3.3 to 5.2) to a milder left skew (about −1). Its fit does nothing, so it cannot leak. feature_names_out='one-to-one' tells the step that output column i is input column i, so get_feature_names_out keeps working after it.

Stateful: a transformer of your own

When the step must learn something, it needs a class with fit and transform. The example is a clipper. total_rooms goes up to 39,320 in a table whose median is about 2,100, and a single extreme value, or a typo in new data, can dominate the distances of the model. The clipper learns, on the training rows only, the 1st and 99th percentiles of each column, and pulls every value outside that range back to the nearest bound:

from sklearn.base import OneToOneFeatureMixin
from sklearn.utils.validation import check_is_fitted, validate_data

class Clipper(OneToOneFeatureMixin, TransformerMixin, BaseEstimator):
    """Clip each column to percentiles learned on the training data."""
    def __init__(self, low=1, high=99):
        self.low = low                     # hyperparameters: stored as given, nothing else
        self.high = high

    def fit(self, X, y=None):
        X = validate_data(self, X)         # to an array; remembers the column names and count
        self.lower_ = np.percentile(X, self.low, axis=0)     # learned state ends with "_"
        self.upper_ = np.percentile(X, self.high, axis=0)
        return self

    def transform(self, X):
        check_is_fitted(self)
        X = validate_data(self, X, reset=False)             # same columns as in fit, or an error
        return np.clip(X, self.lower_, self.upper_)

clipped = make_pipeline(
    ColumnTransformer([
        ('num', make_pipeline(SimpleImputer(strategy='median'), Clipper(), StandardScaler()), num),
        ('cat', OneHotEncoder(handle_unknown='ignore', sparse_output=False), ['ocean_proximity'])]),
    KNeighborsRegressor(n_neighbors=10))
clipped.fit(X_train, y_train)
clip = clipped[0].named_transformers_['num'][1]
print(pd.DataFrame({'lower_': clip.lower_, 'upper_': clip.upper_}, index=num).round(1))
print("MAE:", mae(clipped.predict(X_test)))

                    lower_   upper_
longitude           -123.2   -116.3
latitude              32.7     40.6
housing_median_age     4.0     52.0
total_rooms          162.2  11248.0
total_bedrooms        35.0   2245.6
population            88.0   5825.3
households            30.0   2006.6
median_income          1.1     10.7
MAE: $41,361

The conventions are what make a class a scikit-learn step:

• __init__ only stores the hyperparameters, under the same names as the arguments, and does nothing else. BaseEstimator reads them from there to provide get_params, set_params and the printed representation.
• fit learns and returns self. Everything learned from data gets a name that ends with an underscore (lower_, upper_), the same convention as mean_ and statistics_. validate_data converts the input to an array and records the number and names of the columns.
• transform applies what was learned, after checking that the object was fit and that the columns are the same.
• TransformerMixin provides fit_transform (and set_output); OneToOneFeatureMixin provides get_feature_names_out for a step that keeps its columns one to one.

The table is the learned state: the bounds come from the 16,512 training districts. On this test set, the clipper improves the MAE only slightly, from $41,552 to $41,361, because few test values are outside the bounds. Its real job is protection against values nobody has seen. Take the first test district and multiply its total_rooms by 1,000, a unit error of the kind that appears when a new data source starts sending rooms in a different unit:

typo = X_test.head(1).copy()
typo['total_rooms'] *= 1000                # a unit error in one new row
print(f"true value ${y_test.iloc[0]:,.0f} · correct row ${full.predict(X_test.head(1))[0]:,.0f}")
print(f"with the typo: no clipper ${full.predict(typo)[0]:,.0f} · clipper ${clipped.predict(typo)[0]:,.0f}")

true value $65,600 · correct row $90,030
with the typo: no clipper $251,020 · clipper $145,690

Sweeping the multiplier from 1 to 1,000 shows where the difference comes from (figure script):

Up to 11,248 rooms (the learned upper_) the two pipelines almost agree. Beyond it, the unclipped pipeline follows the typo into regions of the data where the only neighbors are huge districts, and its prediction wanders up to $251,020. The clipped one sees at most 11,248 and stays at $145,690. That is still a wrong answer for a district worth $65,600, and no transformer can fix bad input. But the damage is bounded, and the bound was learned from training data only.

8. Looking ahead: tuning through the pipeline

Stage 3. Every setting of every step is reachable from the outside with a double-underscore path, step__parameter, nested as deep as the pipeline is (columntransformer__num__simpleimputer__strategy). get_params lists them, and set_params changes any of them without rebuilding the pipeline:

params = full.get_params()
print(len(params), "parameters, e.g.:")
for name in params:
    if name.endswith(('n_neighbors', 'strategy', 'handle_unknown')):
        print("  ", name, "=", params[name])
full.set_params(kneighborsregressor__n_neighbors=20)
print(full[-1])

46 parameters, e.g.:
   columntransformer__num__simpleimputer__strategy = median
   columntransformer__cat__handle_unknown = ignore
   kneighborsregressor__n_neighbors = 10
KNeighborsRegressor(n_neighbors=20)

After set_params, the next fit uses the new value. This is what makes automatic tuning possible: a search tries several values of kneighborsregressor__n_neighbors or of columntransformer__num__simpleimputer__strategy, and since each try refits the whole pipeline, the preprocessing is re-learned for every candidate on the rows used for training, and never on the rows used for scoring. Choosing such values honestly requires scoring them on data that is neither the training set nor the test set; scikit-learn's GridSearchCV does it with the pipeline and these names, and it is the subject of the cross-validation lesson. Until then, one rule: never choose a setting by looking at the test score.

9. Summary: open, extend, tune

1. A fitted pipeline is a container: named_steps, list indexing and named_transformers_ reach every step, every learned value can be read, and full[:-1] is exactly what the model receives.
2. One file holds the whole chain: joblib.dump saves every step with what it learned, and the loaded pipeline takes raw rows and returns the same predictions.
3. A step with nothing to learn is a FunctionTransformer. A step that learns is a class whose __init__ only stores hyperparameters, whose fit learns attributes ending in _ and returns self, and whose transform applies them.
4. Every setting is reachable as step__parameter, and a search that refits the whole pipeline re-learns the preprocessing on its own training rows for every candidate.

Advanced lab — open, use, save and extend the pipeline

The fundamental lab ended with a pipeline that scores 0.782 on the test passengers and whose learned values all come from the 712 training passengers. This part looks at what that pipeline hands to the model, uses it on new raw passengers, saves it to disk and extends it with a transformer of your own. Continue in the same notebook, in the same session: the steps below use preprocess, model, prep, X_train and X_test from steps 1 to 5.

Step 6 — what the model sees

Predict. How many columns reach the model: the 7 columns the branches use, or more? Will a third-class man have a 1 in one Pclass column or in several?

names = prep.get_feature_names_out()
print(len(names), "columns reach the model:", list(names))
prep.set_output(transform='pandas')
print(prep.transform(X_test.head(3)).round(2).T)

12 columns reach the model: ['num__Age', 'num__SibSp', 'num__Parch', 'fare__Fare', 'cat__Pclass_1', 'cat__Pclass_2', 'cat__Pclass_3', 'cat__Sex_female', 'cat__Sex_male', 'cat__Embarked_C', 'cat__Embarked_Q', 'cat__Embarked_S']
                  565   160   553
num__Age        -0.43  1.11 -0.58
num__SibSp       1.42 -0.47 -0.47
num__Parch      -0.47  0.73 -0.47
fare__Fare       0.28 -0.11 -0.86
cat__Pclass_1    0.00  0.00  0.00
cat__Pclass_2    0.00  0.00  0.00
cat__Pclass_3    1.00  1.00  1.00
cat__Sex_female  0.00  0.00  0.00
cat__Sex_male    1.00  1.00  1.00
cat__Embarked_C  0.00  0.00  1.00
cat__Embarked_Q  0.00  0.00  0.00
cat__Embarked_S  1.00  1.00  0.00

12 columns reach the model: 3 numbers, the fare, and 3 + 2 + 3 one-hot columns. The three test passengers shown are all third-class men. A hand check with the numbers of step 5: the first passenger has two siblings or spouses aboard, \((2 - 0.493)/1.060 = 1.42\); the second travels with one parent or child, \((1 - 0.390)/0.838 = 0.73\); a passenger with no parents or children gets \((0 - 0.390)/0.838 = -0.47\). With set_output, every transformed value has a name and every row keeps its index (565, 160, 553), so any suspicious number can be traced back to the raw row.

Step 7 — new passengers, raw

Predict. One new passenger boarded at a port the training data never saw, and another has no age. Will the pipeline crash, or answer? What number will replace the missing age?

new = pd.DataFrame({
    'PassengerId': [901, 902, 903], 'Pclass': [3, 1, 2],
    'Name': ['Doe, Mr. John', 'Roe, Mrs. Jane', 'Poe, Miss. Ann'],
    'Sex': ['male', 'female', 'female'], 'Age': [25, np.nan, 30],
    'SibSp': [0, 1, 0], 'Parch': [0, 0, 0], 'Ticket': ['A1', 'B2', 'C3'],
    'Fare': [8.05, 80.0, 15.0], 'Cabin': [np.nan, 'C85', np.nan],
    'Embarked': ['S', 'C', 'B'],             # 'B' (Belfast) never appears in the training data
})
print(prep.transform(new)[['num__Age', 'cat__Embarked_C', 'cat__Embarked_Q', 'cat__Embarked_S']].round(2))
print("P(survived):", model.predict_proba(new)[:, 1].round(2), "· prediction:", model.predict(new))

   num__Age  cat__Embarked_C  cat__Embarked_Q  cat__Embarked_S
0     -0.35              0.0              0.0              1.0
1     -0.08              1.0              0.0              0.0
2      0.03              0.0              0.0              0.0
P(survived): [0. 1. 1.] · prediction: [0 1 1]

Three invented passengers, in the raw format of the original table: 11 columns, text included, one missing age and one port that does not exist in the data. The pipeline does not care about the columns it drops (Name, Ticket, Cabin), fills the missing age of the second passenger with the training median, \((28.5 - 29.556)/13.016 = -0.08\), and gives the third, who boarded at "B", zeros in the three Embarked columns instead of an error. The probabilities are the fraction of survivors among the 5 nearest training passengers, so they move in steps of 0.2; here the neighbors are unanimous: the third-class man did not survive, the two women did.

Simulation: an encoder that refuses new categories. Make a copy of model (with clone) whose one-hot encoder of the 'cat' branch has handle_unknown='error'; the path is prep__cat__onehotencoder__handle_unknown, set with set_params. Fit it on the training rows and predict the first two new passengers, then all three. What happens, and what would it mean for a service that answers one request at a time?

from sklearn.base import clone

strict = clone(model).set_params(prep__cat__onehotencoder__handle_unknown='error')
strict.fit(X_train, y_train)
print("first two new passengers:", strict.predict(new.head(2)))
try:
    strict.predict(new)
except ValueError as err:
    print("all three:", str(err).splitlines()[0])

first two new passengers: [0 1]
all three: Found unknown categories ['B'] in column 2 during transform

Step 8 — save, load, compare

Predict. Will the saved file be a few kilobytes or hundreds? Will the loaded model give exactly the same probabilities, or only approximately?

import os
import joblib

joblib.dump(model, 'titanic-knn.joblib')
loaded = joblib.load('titanic-knn.joblib')
print(f"{os.path.getsize('titanic-knn.joblib') / 1e3:.0f} kB")
print("same test predictions:", np.array_equal(loaded.predict(X_test), model.predict(X_test)))
print("same probabilities for the new rows:",
      np.array_equal(loaded.predict_proba(new), model.predict_proba(new)))

162 kB
same test predictions: True
same probabilities for the new rows: True

The file holds the whole chain (imputers, scalers, encoder and classifier, with everything they learned) in 162 kB, most of it the 712 training passengers that k-NN keeps. Loaded back, it gives the same predictions and the same probabilities, to the last digit, from the same raw rows. That file is what a service would load to answer requests. Loading it requires a compatible version of scikit-learn, and loading a file from someone you do not trust is a security risk: both are discussed in MLOps.

Step 9 — a transformer of your own: the title

Predict. Should the list of "frequent" titles be learned on the training rows, or can it be written by hand from all 891 passengers? Will the title improve the accuracy, and are 179 test passengers enough to be sure?

from sklearn.base import BaseEstimator, TransformerMixin

class TitleExtractor(TransformerMixin, BaseEstimator):
    """Name -> title ('Mr', 'Mrs', ...); titles rarer than min_count in TRAIN become 'Rare'."""
    def __init__(self, min_count=10):
        self.min_count = min_count

    def _titles(self, X):
        names = pd.Series(np.asarray(X).ravel())
        return names.str.extract(r',\s*([^.]+)\.')[0].str.strip()

    def fit(self, X, y=None):
        counts = self._titles(X).value_counts()
        self.titles_ = sorted(counts[counts >= self.min_count].index)   # learned on train
        return self

    def transform(self, X):
        t = self._titles(X)
        return t.where(t.isin(self.titles_), 'Rare').to_frame('title')

    def get_feature_names_out(self, input_features=None):
        return np.array(['title'])

prep2 = ColumnTransformer(preprocess.transformers + [
    ('title', make_pipeline(TitleExtractor(), OneHotEncoder(handle_unknown='ignore', sparse_output=False)),
     ['Name'])])
model2 = Pipeline([('prep', prep2), ('knn', KNeighborsClassifier())]).fit(X_train, y_train)
print("titles learned on train:", model2['prep'].named_transformers_['title'][0].titles_)
print("titles of the new rows: ", TitleExtractor().fit(X_train[['Name']]).transform(new[['Name']])['title'].to_list())
print(f"test accuracy · without title {model.score(X_test, y_test):.3f} · with title {model2.score(X_test, y_test):.3f}")

titles learned on train: ['Master', 'Miss', 'Mr', 'Mrs']
titles of the new rows:  ['Mr', 'Mrs', 'Miss']
test accuracy · without title 0.782 · with title 0.816

TitleExtractor has something to learn, so it is a class and not a FunctionTransformer: in fit it counts the titles of the training passengers and keeps those with at least 10 (Master, Miss, Mr, Mrs); in transform, every other title (Dr, Rev, Col, ...) becomes 'Rare'. A title that was frequent only in the test set would be 'Rare' too, which is the honest behavior. get_feature_names_out lets the encoder that follows name its columns. The new branch is added to the three existing ones, and the accuracy goes from 78.2% to 81.6%: "Master" marks boys, and "Miss" often young unmarried women, so the title carries age information even for passengers whose age is missing. A caveat: 3.4 points on 179 passengers is 6 passengers. Deciding to keep the title because of this number would be a decision taken on the test set; the honest way is a validation split (item 3 of Your turn (advanced)) or the cross-validation you will meet later.

Simulation: how rare is "Rare"? min_count is a hyperparameter of TitleExtractor, so the pipeline reaches it as prep__title__titleextractor__min_count. For min_count in 1, 5, 10 and 40, fit a copy of model2 (with clone and set_params) and print the titles it learned and its test accuracy. Which titles survive each threshold, and does the accuracy say which threshold is best?

for k in [1, 5, 10, 40]:
    m = clone(model2).set_params(prep__title__titleextractor__min_count=k).fit(X_train, y_train)
    titles = m['prep'].named_transformers_['title'][0].titles_
    print(f"min_count {k:2d}: {len(titles):2d} titles · test accuracy {m.score(X_test, y_test):.3f} · {titles}")

min_count  1: 14 titles · test accuracy 0.810 · ['Col', 'Don', 'Dr', 'Jonkheer', 'Lady', 'Major', 'Master', 'Miss', 'Mlle', 'Mr', 'Mrs', 'Ms', 'Rev', 'Sir']
min_count  5:  6 titles · test accuracy 0.816 · ['Dr', 'Master', 'Miss', 'Mr', 'Mrs', 'Rev']
min_count 10:  4 titles · test accuracy 0.816 · ['Master', 'Miss', 'Mr', 'Mrs']
min_count 40:  3 titles · test accuracy 0.816 · ['Miss', 'Mr', 'Mrs']

To close the lesson. In one sentence each: (1) why save the whole pipeline and not only the trained classifier? (2) When must a step of your own be a class rather than a FunctionTransformer? (3) Why does a search that tries settings through set_params never let the scoring rows into the preprocessing?

Your turn (advanced)

At home, with more time:

1. Ratios as a step. Rooms per household, bedrooms per room and people per household are more meaningful than raw counts. Write them as a FunctionTransformer (use a function that receives a DataFrame and returns one with the three new columns, and feature_names_out as a function that returns their names), put it in the numeric branch of section 3, and compare the test MAE with the lesson's $41,552.
2. The deck. The first letter of Cabin is the deck (A to G, T), and a missing cabin is information too (mostly third class). Add a branch that turns Cabin into the deck, with 'U' for missing, followed by one-hot encoding. Write the conversion first as a lambda inside a FunctionTransformer, fit, and try joblib.dump on the model. What happens, and why does a function defined with def fix it?
3. Choose k honestly. Carve a validation split out of X_train (25% of it, stratified). For k in 1, 5, 15, 45, use model.set_params(knn__n_neighbors=k), fit on the rest of the training rows and score on the validation rows. Pick the best k, refit on all of X_train, and only then look at the test accuracy, once. (The cross-validation lesson will make this choice more robust.)
4. Clip the fare. Replace log1p in the 'fare' branch with the lesson's Clipper (percentiles 1 and 99). Print its upper_ and compare it with the largest fare in the training rows. Does the test accuracy change? Would you expect k-NN to be more sensitive to the fare's outliers than a model that only compares a value to a threshold?

References

• Buitinck, L.; Louppe, G.; Blondel, M.; Pedregosa, F.; Mueller, A.; Grisel, O.; Niculae, V.; Prettenhofer, P.; Gramfort, A.; Grobler, J.; Layton, R.; VanderPlas, J.; Joly, A.; Holt, B.; Varoquaux, G. "API design for machine learning software: experiences from the scikit-learn project." ECML PKDD Workshop: Languages for Data Mining and Machine Learning (2013): 108–122 — the estimator, transformer and pipeline interfaces used in this lesson, and why they look the way they do. arXiv
• Pedregosa, F.; Varoquaux, G.; Gramfort, A.; Michel, V.; Thirion, B.; Grisel, O.; Blondel, M.; Prettenhofer, P.; Weiss, R.; Dubourg, V.; Vanderplas, J.; Passos, A.; Cournapeau, D.; Brucher, M.; Perrot, M.; Duchesnay, É. "Scikit-learn: Machine Learning in Python." Journal of Machine Learning Research 12 (2011): 2825–2830. full text
• Micci-Barreca, D. "A Preprocessing Scheme for High-Cardinality Categorical Attributes in Classification and Prediction Problems." ACM SIGKDD Explorations 3, no. 1 (2001): 27–32 — target encoding with smoothing, the method behind TargetEncoder. DOI
• Kaufman, S.; Rosset, S.; Perlich, C.; Stitelman, O. "Leakage in Data Mining: Formulation, Detection, and Avoidance." ACM TKDD 6, no. 4 (2012) — a taxonomy of leakage with real competition cases. DOI
• Sculley, D.; Holt, G.; Golovin, D.; Davydov, E.; Phillips, T.; Ebner, D.; Chaudhary, V.; Young, M.; Crespo, J.-F.; Dennison, D. "Hidden Technical Debt in Machine Learning Systems." Advances in Neural Information Processing Systems 28 (NeurIPS 2015): 2503–2511 — "pipeline jungles" and the cost of training code and serving code that drift apart. full text

Quiz

The quiz covers the fundamental approach.