Data Preprocessing

Raw data is almost never ready for a learning algorithm. Features live on different scales, categories are stored as strings, values are missing, and distributions are skewed. Preprocessing transforms raw features into a representation the algorithm can exploit — and done wrong, it is also the easiest place to leak information (see Validation & Data Leakage).

Why scale features?

Many algorithms are distance-based or gradient-based, and both are distorted when features have wildly different magnitudes:

• k-NN computes Euclidean distances: a feature ranging in the thousands (income) drowns out one ranging in the tens (age);
• Gradient descent converges slowly on elongated loss surfaces created by unscaled features;
• SVMs and regularized models (Ridge/Lasso) penalize coefficients as if features were comparable;
• PCA finds directions of maximum variance — the largest-scale feature wins by default.

Tree-based models (decision trees, random forests, gradient boosting) are the notable exception: they split on thresholds, so monotonic transformations do not affect them.

Scaling methods

Standardization (z-score): center at zero, unit variance —

Min-max normalization: squeeze into \([0, 1]\) —

Robust scaling: use median and IQR instead of mean and standard deviation —

The choice matters when outliers are present. Min-max is fully determined by the two most extreme points; standardization is somewhat distorted by them; robust scaling ignores them:

from sklearn.preprocessing import StandardScaler

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # learn μ, σ on TRAIN only
X_test_scaled = scaler.transform(X_test)        # apply the SAME μ, σ

Skewed features

Right-skewed features (income, prices, counts) often benefit from a log transform, \(x' = \log(1 + x)\), which compresses the long tail and makes the distribution more symmetric. More general tools: PowerTransformer (Box-Cox, Yeo-Johnson) and QuantileTransformer.

Encoding categorical features

Algorithms consume numbers, not strings. The two workhorses:

One-hot encoding — one binary column per category:

from sklearn.preprocessing import OneHotEncoder
enc = OneHotEncoder(handle_unknown='ignore')  # unseen categories → all zeros

• Safe for nominal categories (no order): city, color, product type;
• Explodes dimensionality for high-cardinality features (zip codes → thousands of columns).

Ordinal encoding — map categories to integers:

from sklearn.preprocessing import OrdinalEncoder
enc = OrdinalEncoder(categories=[['small', 'medium', 'large']])

• Correct for ordinal categories (small < medium < large);
• Wrong for nominal ones: encoding cities as São Paulo=0, Rio=1, Recife=2 invents an order and distances that do not exist — linear models and k-NN will happily exploit the fiction.

For high-cardinality features, consider target encoding (replace each category by a smoothed mean of the target) — powerful but leakage-prone: it must be fit inside cross-validation.

Missing value imputation

Building on the EDA discussion of missingness mechanisms:

from sklearn.impute import SimpleImputer, KNNImputer

SimpleImputer(strategy='median')       # numeric: robust default
SimpleImputer(strategy='most_frequent')  # categorical
KNNImputer(n_neighbors=5)              # impute from similar rows

Two useful practices:

• add a missing-indicator column (add_indicator=True) — the fact that a value was missing is often predictive;
• impute inside a Pipeline, so the imputation statistics are learned from training folds only.

The preprocessing map

flowchart TD
    A[Raw feature] --> B{Type?}
    B -->|numeric| C{Skewed?}
    C -->|yes| D[log / power transform] --> E
    C -->|no| E{Outliers?}
    E -->|yes| F[RobustScaler]
    E -->|no| G[StandardScaler]
    B -->|categorical| H{Ordered?}
    H -->|yes| I[OrdinalEncoder]
    H -->|no| J{Cardinality?}
    J -->|low| K[OneHotEncoder]
    J -->|high| L[target encoding / grouping]

Class materials

Quiz