Machine Learning · Insper

Pipelines

Advanced approach

Open the pipeline, save it, extend it with steps of your own and tune it by name

The path of this part

Open and savenamed steps, learned values, one file
Steps of your ownFunctionTransformer and a class that learns
Tune through itstep__parameter and set_params

Each stage answers one question: what did each step learn, and how do we ship it? → how do we add a step scikit-learn does not have? → how does a search reach every setting?

Stage 1 · What did each step learn, and how do we ship it?

A fitted pipeline is a container you can open


list(full.named_steps)
full[-1]                   # indexing, like a list
full.named_steps['columntransformer'] \
    .named_transformers_['num'][0].statistics_
full[:-1].transform(X_test.head(2)).shape


['columntransformer', 'kneighborsregressor']
KNeighborsRegressor(n_neighbors=10)
[-118.5 34.2 29. 2129. 437. 1169. 411. 3.5]
(2, 13)

named_transformers_ opens a branch of the ColumnTransformer: the eight training medians the imputer will use (437 bedrooms).
A slice is a smaller pipeline: full[:-1] is the preprocessing only, exactly what the model receives.
print(full) gives the text diagram; in Jupyter the pipeline is drawn as a clickable diagram.

One file: preprocessing + model


joblib.dump(full, 'housing-knn.joblib')
loaded = joblib.load('housing-knn.joblib')
np.array_equal(loaded.predict(X_test), full.predict(X_test))   # True

3.9 MB on disk

k-NN (k-nearest neighbors) keeps all 16,512 training districts to find neighbors; most models store far less.

identical predictions

The loaded pipeline takes raw rows and returns the same predictions, to the last digit.

Whoever loads it cannot forget a step, re-fit a scaler or reorder the columns. Versions and safe loading are the subject of the lesson on MLOps (machine-learning operations).

Stage 2 · How do we add a step scikit-learn does not have?

Your own steps: stateless


log = FunctionTransformer(np.log1p, feature_names_out='one-to-one')   # nothing to learn

	skewness of the four count columns
before	3.85, 3.32, 5.17, 3.26
after $\log(1+x)$	−1.09, −1.0, −1.04, −1.06

Its fit does nothing, so it cannot leak. 'one-to-one' keeps get_feature_names_out working.

Your own steps: a transformer that learns

A clipper: learn the 1st and 99th percentiles of each column on the training rows, pull every value outside them back to the bound.

__init__ only stores the hyperparameters.
fit learns and returns self; learned state ends with _ (lower_, upper_).
transform applies what was learned, after checking it was fit.
TransformerMixin gives fit_transform; OneToOneFeatureMixin the feature names.

learned on 16,512 districts

total_rooms: 162.2 to 11,248.0

$41,361

mean absolute error (MAE), slightly better than $41,552

A transformer that learns: the code


class Clipper(OneToOneFeatureMixin, TransformerMixin, BaseEstimator):
    def __init__(self, low=1, high=99):
        self.low = low                     # hyperparameters: stored, nothing else
        self.high = high

    def fit(self, X, y=None):
        X = validate_data(self, X)         # remembers the column names and count
        self.lower_ = np.percentile(X, self.low, axis=0)   # learned state ends with "_"
        self.upper_ = np.percentile(X, self.high, axis=0)
        return self

    def transform(self, X):
        check_is_fitted(self)
        X = validate_data(self, X, reset=False)            # same columns as in fit
        return np.clip(X, self.lower_, self.upper_)

It goes in the numeric branch, between the imputer and the scaler: make_pipeline(SimpleImputer(strategy='median'), Clipper(), StandardScaler()).

What the clipper is really for

Predict. One new district arrives with its total_rooms multiplied by 1,000 (a unit error). What happens to its prediction, with and without the clipper?

true value	$65,600
correct row	$90,030
typo, no clipper	$251,020
typo, clipper	$145,690

Predicted value of one district as total_rooms is multiplied from 1 to 1000: without the clipper it climbs to $251k; with it, it stops at $146k

Still wrong, and no transformer can fix bad input. But the damage is bounded, and the bound was learned from training data only.

Stage 3 · How does a search reach every setting?

Looking ahead: tuning through the pipeline


full.get_params()        # 46 parameters, e.g. columntransformer__num__simpleimputer__strategy
full.set_params(kneighborsregressor__n_neighbors=20)

Every setting of every step is reachable as step__parameter.
A search that refits the whole pipeline re-learns the preprocessing on the training rows of each try.

Never choose a setting by looking at the test score: the cross-validation lesson does it honestly, with GridSearchCV and these names.

Summary

A fitted pipeline is a container: named_steps, indexing, named_transformers_; every learned value can be read.
One file holds the whole chain; the loaded pipeline takes raw rows.
Nothing to learn: FunctionTransformer. Something to learn: a class whose fit stores attributes ending in _.
Every setting is reachable as step__parameter; a search refits the whole pipeline.

Design habit: write the pipeline skeleton first. The object you evaluate is the object you ship.

Advanced lab · about 30 min, in pairs

Open, use, save and extend the pipeline, steps 6 to 9

Same notebook, same session: the steps use preprocess, model, prep, X_train and X_test from steps 1 to 5.
What the model sees (set_output); new raw passengers; save, load, compare; a transformer of your own: the title.
Lab notebook, from step 6.
Simulations: an encoder that refuses new categories (step 7) and how rare is "Rare"? (step 9).

Pace: steps 6 and 7 and the first simulation in 12 min; step 8 in 5 min; step 9 and the second in 13 min.

	skewness of the four count columns
before	3.85, 3.32, 5.17, 3.26
after \(\log(1+x)\)	−1.09, −1.0, −1.04, −1.06