Data Preprocessing

The algorithm only sees numbers

It minimizes the cost with the numbers you hand it. If they are badly prepared, it minimizes the wrong thing, with full confidence.

The plan: scaling → shape → encoding → imputation, each measured on a small example, then on the Titanic (891 passengers of the 1912 voyage). EDA looked at the table; this lesson changes it.

The path of this lesson

Three passengers, by hand

In raw units, the 70-year-old wins

48.5 years is a smaller number than 59.35 pounds. Nothing says a year and a pound should weigh the same: it is just what happens when you subtract raw numbers.

Standardized, the ranking flips

A–B: \(48.5 / 14.52 = \mathbf{3.34}\) std

A–C: \(59.35 / 52.88 = \mathbf{1.12}\) std

\(\mu\) and \(\sigma\) computed over the 714 passengers whose age is known.

Simulator: the unit decides the neighbors

Predict. Slide the income unit from thousands to cents. Do the 5 nearest neighbors of the red point stay the same? Then press standardize.

Which column owns the distance?

For two random rows, the expected squared difference in a column is twice its variance: each column's share of the distance is its share of the total variance.

Which models care about scale

Four scalers: \(x' = (x - \text{center}) / \text{spread}\)

By hand: a column with an outlier

Five households, monthly income in R$ thousand: 1, 2, 3, 4 and 40.

• mean \(\mu = 10\); \(\sigma = \sqrt{(81+64+49+36+900)/5} = \sqrt{226} = 15.03\) (scikit-learn divides by \(n\))
• minimum 1, maximum 40, range 39
• median 3; \(Q_1 = 2\), \(Q_3 = 4\), so IQR = 2

First household (income 1): standard \((1-10)/15.03 = -0.599\) · min-max \(0\) · robust \((1-3)/2 = -1\) · max-abs \(1/40 = 0.025\)

The outlier sets the scale

Learned statistics live in attributes ending with an underscore: mean_, scale_, center_.

Same geometry, different weight next to other columns

Inside the column the ordinary households always occupy the same 8% of the range: a linear map cannot change proportions. What changes is how big their differences are compared with the other columns, and that is what a distance adds up.

Choosing a scaler

• Normalization is ambiguous (min-max? "make it normal"? scikit-learn's Normalizer rescales each row). Write the formula.
• A new value outside the training range leaves the interval: an income of 80 becomes \((80-1)/39 = 2.03\) under min-max.

Fit on the training set, transform everything else

A mean or a standard deviation is a parameter of your model, just like a weight. fit learns and stores, transform applies, fit_transform does both on the same data.

Four situations

Skewed features: a scaler cannot change the shape

Titanic fares: half cost less than £14.46, three cost £512, and 15 tickets were free. Skewness \(= \frac{1}{n}\sum z_i^3\): 0 when symmetric, positive with a long right tail.

Let the data choose the power

Encoding: three kinds of categories

Integer codes invent distances

Ordinal: you declare the order

OrdinalEncoder().fit(levels).categories_
# ['bachelor', 'high school', 'master']   alphabetical!
OrdinalEncoder(categories=[['high school', 'bachelor', 'master']])
# codes now follow the real order: 0, 1, 2
  

The alphabet puts high school between bachelor and master.

One-hot: two practical options

High cardinality, and target encoding

Ticket has 681 distinct values for 891 rows: one-hot would create 681 columns of almost only zeros, a recipe for overfitting.

• First look for the structure inside: the deck letter of Cabin, the title in Name (Mr, Miss, Mrs, Master...).
• Then group rare levels: min_frequency=10 keeps 4 titles and merges the other 13.
• Target encoding: replace each category by the mean of the target in it. One column, whatever the cardinality.

Target encoding leaks

• Most tickets appear once or twice, so a row's encoded value is mostly its own label: the model learns to read the answer, not to predict it.
• Cross-fitting: fit_transform encodes each fold with means from the other folds, so no row sees its own label; rare categories are shrunk toward the global mean.

Missing values: why is it missing?

Filling with one value shrinks the variance

Six values, two missing: 2, 4, ?, 6, ?, 8. Observed mean 5, variance \((9+1+1+9)/4 = 5\).

Imputing from the other columns

Keep the fact that it was missing

imp = SimpleImputer(strategy='median', add_indicator=True)
imp.get_feature_names_out()   # ['Age' 'missingindicator_Age']
# passenger 1: [22, 0]    passenger 6: [28, 1]
  

Missing age goes with lower survival (29.4% against 40.6%). The flag costs one column and lets the model decide whether missingness is informative. For categories: a category of its own, 'unknown'.

The preprocessing map

Per column: missing values → shape → scale, every step fitted on the training rows. Click the answers to build a recipe.

Which families care about which step

Lab: preparing the Titanic for a model

1. Inventory: Survived is the target, PassengerId goes out, Pclass is a category stored as a number.
2. Split first: 712 train, 179 test, stratified.
3. Impute Age (median), encode Sex, Embarked, Pclass (one-hot), scale the numbers.
4. All in one ColumnTransformer: 12 columns, no NaN, no text.

Lab notebook (Jupyter or Colab, File → Upload notebook).

Same model, same data, different scaler

Summary

1. Raw units decide distances by accident: scale what distances, variances and gradients use.
2. Scalers differ in center and spread; outliers set the scale except for RobustScaler.
3. A scaler cannot fix shape: log1p, Yeo-Johnson, then scale.
4. One-hot for nominal, declared order for ordinal; target encoding only cross-fitted.
5. Ask why it is missing; imputation shrinks spread; keep a flag.
6. Fit on train, transform everything else.