Data Preprocessing

Raw data is almost never ready for a learning algorithm. Features live on different scales, categories are stored as strings, values are missing, and distributions are skewed. Preprocessing transforms raw features into a representation the algorithm can exploit — and done wrong, it is also the easiest place to leak information (see Validation & Data Leakage).

Why scale features?

Many algorithms are distance-based or gradient-based, and both are distorted when features have wildly different magnitudes:

• k-NN computes Euclidean distances: a feature ranging in the thousands (income) drowns out one ranging in the tens (age);
• Gradient descent converges slowly on elongated loss surfaces created by unscaled features;
• SVMs and regularized models (Ridge/Lasso) penalize coefficients as if features were comparable;
• PCA finds directions of maximum variance — the largest-scale feature wins by default.

Tree-based models (decision trees, random forests, gradient boosting) are the notable exception: they split on thresholds, so monotonic transformations do not affect them.

Scaling methods

Standardization (z-score): center at zero, unit variance —

Min-max normalization: squeeze into \([0, 1]\) —

Robust scaling: use median and IQR instead of mean and standard deviation —

The choice matters when outliers are present. Min-max is fully determined by the two most extreme points; standardization is somewhat distorted by them; robust scaling ignores them:

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # learn μ, σ on TRAIN only
X_test_scaled = scaler.transform(X_test)        # apply the SAME μ, σ

Skewed features

Right-skewed features (income, prices, counts) often benefit from a log transform, \(x' = \log(1 + x)\), which compresses the long tail and makes the distribution more symmetric. More general tools: PowerTransformer (Box-Cox, Yeo-Johnson) and QuantileTransformer.

Encoding categorical features

Algorithms consume numbers, not strings. The two workhorses:

One-hot encoding — one binary column per category:

from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(handle_unknown='ignore')  # unseen categories → all zeros

• Safe for nominal categories (no order): city, color, product type;
• Explodes dimensionality for high-cardinality features (zip codes → thousands of columns).

Ordinal encoding — map categories to integers:

from sklearn.preprocessing import OrdinalEncoder
enc = OrdinalEncoder(categories=[['small', 'medium', 'large']])

• Correct for ordinal categories (small < medium < large);
• Wrong for nominal ones: encoding cities as São Paulo=0, Rio=1, Recife=2 invents an order and distances that do not exist — linear models and k-NN will happily exploit the fiction.

For high-cardinality features, consider target encoding (replace each category by a smoothed mean of the target) — powerful but leakage-prone: it must be fit inside cross-validation.

Missing value imputation

Building on the EDA discussion of missingness mechanisms:

from sklearn.impute import SimpleImputer, KNNImputer

SimpleImputer(strategy='median')       # numeric: robust default
SimpleImputer(strategy='most_frequent')  # categorical
KNNImputer(n_neighbors=5)              # impute from similar rows

Two useful practices:

• add a missing-indicator column (add_indicator=True) — the fact that a value was missing is often predictive;
• impute inside a Pipeline, so the imputation statistics are learned from training folds only.

The preprocessing map

flowchart TD
    A[Raw feature] --> B{Type?}
    B -->|numeric| C{Skewed?}
    C -->|yes| D[log / power transform] --> E
    C -->|no| E{Outliers?}
    E -->|yes| F[RobustScaler]
    E -->|no| G[StandardScaler]
    B -->|categorical| H{Ordered?}
    H -->|yes| I[OrdinalEncoder]
    H -->|no| J{Cardinality?}
    J -->|low| K[OneHotEncoder]
    J -->|high| L[target encoding / grouping]

What a unit change actually does

The argument for scaling is usually made in the abstract. Here it is as a concrete consequence: the same forty people, the same two columns, and the only thing that changes is the unit printed on one of them.

Slide the unit from thousands to cents and watch the five nearest neighbours of the red point become a different five. Nothing about anyone changed. But k-NN would now return a different answer, k-means would find different clusters, and PCA would aim its first component at whichever column happens to carry bigger numbers.

That is the precise sense in which distance-based and gradient-based methods are "sensitive to scale": the algorithm is not reading your features, it is reading their numerical magnitudes. Press standardise and the neighbour list stops depending on the unit at all — which is the whole point of the operation.

Class materials

References

• Box, G. E. P.; Cox, D. R. "An Analysis of Transformations." JRSS B 26 (1964). DOI

The full course bibliography is on the references page.

Quiz