Skip to content

Pipelines

A real preprocessing chain — impute, encode, scale, then model — is a sequence of steps that must be fit on training data only and replayed identically on validation folds, the test set, and production traffic. Doing this by hand invites bugs. scikit-learn's Pipeline packages the whole chain as a single estimator.

The problem pipelines solve

Without a pipeline, the honest workflow requires careful bookkeeping:

# Fragile: every step must be manually fit on train, applied to test
imputer.fit(X_train)
X_train_i = imputer.transform(X_train)
X_test_i = imputer.transform(X_test)

scaler.fit(X_train_i)
X_train_s = scaler.transform(X_train_i)
X_test_s = scaler.transform(X_test_i)

model.fit(X_train_s, y_train)
model.predict(X_test_s)

One misplaced fit — or a fit_transform on the full dataset — and you have data leakage. Worse, during cross-validation the preprocessing must be re-fit on each training fold, which is practically impossible to do correctly by hand.

Pipeline: one estimator, many steps

from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import StandardScaler
from sklearn.linear_model import LogisticRegression

pipe = Pipeline([
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler()),
    ('model', LogisticRegression()),
])

pipe.fit(X_train, y_train)      # fits imputer → scaler → model, in order, on train
pipe.predict(X_test)            # transforms X_test through the SAME fitted steps

Rules of the composition:

  • every step except the last must be a transformer (fit + transform);
  • the last step is typically an estimator (fit + predict);
  • pipe.fit(X, y) calls fit_transform on each transformer in sequence, then fit on the estimator;
  • pipe.predict(X) calls only transform on each transformer — parameters are frozen.

Because the pipeline is an estimator, it drops directly into cross_val_score and GridSearchCV — and preprocessing is automatically re-fit inside each fold, killing the leakage bug by construction:

from sklearn.model_selection import cross_val_score
scores = cross_val_score(pipe, X, y, cv=5)   # leak-free by design

Heterogeneous columns: ColumnTransformer

Real tables mix numeric and categorical columns that need different treatments. ColumnTransformer routes column subsets to parallel preprocessing branches and concatenates the results:

from sklearn.compose import ColumnTransformer
from sklearn.preprocessing import OneHotEncoder

numeric = ['age', 'income', 'tenure']
categorical = ['city', 'plan', 'device']

preprocess = ColumnTransformer([
    ('num', Pipeline([
        ('imputer', SimpleImputer(strategy='median')),
        ('scaler', StandardScaler()),
    ]), numeric),
    ('cat', Pipeline([
        ('imputer', SimpleImputer(strategy='most_frequent')),
        ('onehot', OneHotEncoder(handle_unknown='ignore')),
    ]), categorical),
])

pipe = Pipeline([
    ('preprocess', preprocess),
    ('model', LogisticRegression(max_iter=1000)),
])

flowchart LR
    X[Raw DataFrame] --> CT{ColumnTransformer}
    CT -->|numeric cols| N[median impute → scale]
    CT -->|categorical cols| C[mode impute → one-hot]
    N --> J[concatenate]
    C --> J
    J --> M[LogisticRegression]

Tuning through the pipeline

Hyperparameters of any step are addressed as stepname__paramname (double underscore) — so a single grid search can tune preprocessing choices and model hyperparameters together, honestly:

from sklearn.model_selection import GridSearchCV

param_grid = {
    'preprocess__num__imputer__strategy': ['mean', 'median'],
    'model__C': [0.01, 0.1, 1, 10],
}
search = GridSearchCV(pipe, param_grid, cv=5, scoring='f1')
search.fit(X_train, y_train)

More on this in Model Selection.

Inspection and persistence

pipe.named_steps['model'].coef_            # access any fitted step
pipe[:-1].transform(X_train)               # run preprocessing only
pipe.get_feature_names_out()               # names after one-hot expansion

import joblib
joblib.dump(pipe, 'churn-model.joblib')    # ship ONE artifact: preprocessing + model

Persisting the whole pipeline is the foundation of reliable MLOps: production code cannot "forget" a preprocessing step, because the steps travel inside the artifact.

Design habit

Start every project by writing the pipeline skeleton — even before choosing the model. It forces the train/test discipline from the first line of code and makes every later experiment a one-line change.

Class materials

Class notebook (in Portuguese)

Hands-on notebook used in class — Aula 04 — Pipelines: open in Colab

Quiz