Overview

Preprocessing

Everything on this page happens on the right of the split. That is not a filing convention: every step here estimates something from the rows it is given — a mean, a median, a category list, a projection basis — and estimating it from rows you will later be scored on is what the leakage chapter is about.

So there is one structural rule, and then a set of choices.

The rule

Everything with a fit goes inside the Pipeline, and the Pipeline is what you cross-validate.

from sklearn.compose import ColumnTransformer
from sklearn.pipeline import Pipeline
from sklearn.impute import SimpleImputer
from sklearn.preprocessing import OneHotEncoder, StandardScaler

numeric = Pipeline([('impute', SimpleImputer(strategy='median')),
                    ('scale',  StandardScaler())])
categorical = Pipeline([('impute', SimpleImputer(strategy='most_frequent')),
                        ('encode', OneHotEncoder(handle_unknown='ignore'))])

prep = ColumnTransformer([('num', numeric, numeric_columns),
                          ('cat', categorical, categorical_columns)])

model = Pipeline([('prep', prep), ('net', MLPClassifier())])
cross_val_score(model, X, y, cv=5)      # every fit re-runs inside every fold

Doing it by hand is correct exactly once, for one split. The moment you cross-validate or tune, the hand-written version is fitted once while the folds change five times — and every fold is contaminated again.

The four questions

Scaling & Normalization

The columns are in incomparable units, so the first gradient is about the measurement system. Which scaler, and what one typo does to each of them.

Scaling
Transforming Shape

Scaling cannot change a distribution's shape. When the shape is the problem: log, power, quantile, binning — and the criterion that is not a symmetric histogram.

Transforms
Encoding

Turning categories into numbers is asserting which comparisons between them are meaningful. Covered with the taxonomy, on the feature types page.

Feature types
The Curse of Dimensionality

What high dimension actually does to distances, volumes and angles — four separate phenomena, only one of which depends on your data. The reason the next page exists.

The curse
Dimensionality Reduction

PCA, t-SNE and UMAP answer three different questions and lose three different things. Which distortion each one introduces, and how to judge a projection.

Reduction

Order of operations

Within the pipeline, the order is not arbitrary either:

flowchart LR
    A["impute"] --> B["encode<br/><small>categoricals</small>"]
    B --> C["transform shape<br/><small>log, power</small>"]
    C --> D["scale"]
    D --> E["reduce<br/><small>optional</small>"]
    E --> F["model"]

    classDef s fill:#eef2f7,stroke:#8b949e,color:#1f2937
    class A,B,C,D,E,F s

Impute first, because everything after it breaks on NaN.
Transform before scaling, because a log changes the mean and the standard deviation that the scaler is about to estimate.
Scale before reducing, because PCA maximizes variance and variance has units — unstandardized, the column with the largest units becomes the first principal component by construction.

Two things that look like preprocessing and are not

Resampling for class imbalance is not a transformation of the features — it changes the rows, and it must happen inside the training fold only, never on the validation rows. sklearn's Pipeline will not do this correctly; imblearn.pipeline.Pipeline will. See class imbalance.

Cleaning — dropping duplicates, fixing a known unit error, replacing a sentinel with NaN — estimates nothing, so it belongs before the split, where it is also safer. See data quality.

A worked example

The Titanic dataset, as it arrives:

PassengerId	Survived	Pclass	Name	Sex	Age	SibSp	Parch	Ticket	Fare	Cabin	Embarked
710	1	3	Moubarek, Master. Halim Gonios ("William George")	male	nan	1	1	2661	15.2458	nan	C
440	0	2	Kvillner, Mr. Johan Henrik Johannesson	male	31	0	0	C.A. 18723	10.5	nan	S
841	0	3	Alhomaki, Mr. Ilmari Rudolf	male	20	0	0	SOTON/O2 3101287	7.925	nan	S
721	1	2	Harper, Miss. Annie Jessie "Nina"	female	6	0	1	248727	33	nan	S
40	1	3	Nicola-Yarred, Miss. Jamila	female	14	1	0	2651	11.2417	nan	C
291	1	1	Barber, Miss. Ellen "Nellie"	female	26	0	0	19877	78.85	nan	S
301	1	3	Kelly, Miss. Anna Katherine "Annie Kate"	female	nan	0	0	9234	7.75	nan	Q
334	0	3	Vander Planke, Mr. Leo Edmondus	male	16	2	0	345764	18	nan	S
209	1	3	Carr, Miss. Helen "Ellen"	female	16	0	0	367231	7.75	nan	Q
137	1	1	Newsom, Miss. Helen Monypeny	female	19	0	2	11752	26.2833	D47	S

Ten rows of the raw table: mixed types, missing ages, a categorical port of embarkation.

ResultCodeEncodedCode

Pclass	Sex	Age	SibSp	Parch	Fare	Embarked
3	male	17.5	1	1	15.2458	C
2	male	31	0	0	10.5	S
3	male	20	0	0	7.925	S
2	female	6	0	1	33	S
3	female	14	1	0	11.2417	C
1	female	26	0	0	78.85	S
3	female	17.5	0	0	7.75	Q
3	male	16	2	0	18	S
3	female	16	0	0	7.75	Q
1	female	19	0	2	26.2833	S

import pandas as pd

# Preprocess the data
def preprocess(df):
    # Fill missing values
    df['Age'] = df['Age'].fillna(df['Age'].median())
    df['Embarked'] = df['Embarked'].fillna(df['Embarked'].mode()[0])
    df['Fare'] = df['Fare'].fillna(df['Fare'].median())

    # Select features
    features = ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked']
    return df[features]

# Load the Titanic dataset
df = pd.read_csv('https://raw.githubusercontent.com/hsandmann/ml/refs/heads/main/data/kaggle/titanic-dataset.csv')
df = df.sample(n=10, random_state=42)

# Preprocessing
df = preprocess(df)

# Display the first few rows of the dataset
print(df.to_markdown(index=False))

Pclass	Sex	Age	SibSp	Parch	Fare	Embarked
3	0	16	0	0	7.75	1
2	1	31	0	0	10.5	2
1	0	26	0	0	78.85	2
3	1	17.5	1	1	15.2458	0
3	1	16	2	0	18	2
3	1	20	0	0	7.925	2
1	0	19	0	2	26.2833	2

import pandas as pd
from sklearn.preprocessing import LabelEncoder

# Preprocess the data
def preprocess(df):
    # Fill missing values
    df['Age'] = df['Age'].fillna(df['Age'].median())
    df['Embarked'] = df['Embarked'].fillna(df['Embarked'].mode()[0])
    df['Fare'] = df['Fare'].fillna(df['Fare'].median())

    # Convert categorical variables
    label_encoder = LabelEncoder()
    df['Sex'] = label_encoder.fit_transform(df['Sex'])
    df['Embarked'] = label_encoder.fit_transform(df['Embarked'])

    # Select features
    features = ['Pclass', 'Sex', 'Age', 'SibSp', 'Parch', 'Fare', 'Embarked']
    return df[features]

# Load the Titanic dataset
df = pd.read_csv('https://raw.githubusercontent.com/hsandmann/ml/refs/heads/main/data/kaggle/titanic-dataset.csv')
df = df.sample(n=10, random_state=42)

# Preprocessing
df = preprocess(df)

# Display the first few rows of the dataset
print(df.sample(n=7, random_state=42).to_markdown(index=False))

Read that code as an illustration, not as a template

Both scripts compute their statistics on the whole table, because they are printing a table rather than training a model. In a real pipeline the median that fills Age and the categories that build the one-hot block are both fit steps, and they belong inside the Pipeline above — refitted on every fold.