Scaling & Normalization

Scaling and Normalization

A network's first layer computes \(\sum_j w_j x_j\). If \(x_1\) is measured in the hundred thousands and \(x_2\) in the thousandths, then at initialization — when all the \(w_j\) are drawn from the same small distribution — the sum is entirely \(x_1\), the gradient is entirely about \(x_1\), and \(x_2\) effectively does not exist yet.

That is the whole argument. Scaling is not cosmetic and it is not about "making the data nice". It is about making the units of the inputs comparable, so that the optimizer's first steps are about the problem rather than about the measurement system.

"""Six columns on six different scales, and a network that cannot cope.

The columns are the same standard normal draw, multiplied by 1, 1000, 0.001,
50, 100000 and 5. The label depends on all six equally — in the *unscaled*
sense of equally, which is the point: whatever the units, each column carries
the same amount of information about y.

A network does not know that. It sees one input in the hundred thousands and
one in the thousandths, and the first gradient step is dominated entirely by
the former. The table is what that costs, and what any of the standard scalers
buys back.

Printed as a markdown table, in identifiers only, so one artifact serves both
the English and the Portuguese page.
"""

import numpy as np
from sklearn.model_selection import StratifiedKFold, cross_val_score
from sklearn.neural_network import MLPClassifier
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import (MaxAbsScaler, MinMaxScaler, QuantileTransformer,
                                   RobustScaler, StandardScaler)

N, DIM = 4000, 6
UNITS = np.array([1.0, 1000.0, 0.001, 50.0, 100_000.0, 5.0])

rng = np.random.default_rng(31)
Z = rng.normal(size=(N, DIM))                 # the honest, unit-free measurements
X = Z * UNITS                                 # the same thing, as it arrives in the table
weights = rng.normal(size=DIM)
y = ((Z @ weights + 0.8 * rng.normal(size=N)) > 0).astype(int)

net = lambda: MLPClassifier(hidden_layer_sizes=(32, 16), max_iter=600, random_state=0)
cv = StratifiedKFold(5, shuffle=True, random_state=0)

scalers = {
    "no_scaling": None,
    "StandardScaler": StandardScaler(),
    "MinMaxScaler": MinMaxScaler(),
    "RobustScaler": RobustScaler(),
    "MaxAbsScaler": MaxAbsScaler(),
    "QuantileTransformer": QuantileTransformer(
        output_distribution="normal", n_quantiles=500, random_state=0),
}

print("| `preprocessing` | `accuracy` |")
print("|---|---:|")
for name, scaler in scalers.items():
    model = net() if scaler is None else make_pipeline(scaler, net())
    print(f"| `{name}` | **{cross_val_score(model, X, y, cv=cv).mean():.3f}** |")

Six columns, all carrying exactly the same amount of information about the label, multiplied by 1, 1000, 0.001, 50, 100 000 and 5. Unscaled, the network reaches 0.605. With any scaler at all it reaches 0.89–0.91. The 30 points were not a modelling problem; they were a units problem.

The five that matter

Read the right-hand column, because it is the one that predicts how each behaves. Every parameter in it is either a moment or an extreme, and moments and extremes respond very differently to a bad value.

Slide the typo up from nothing and watch the honest data — every row except the bad one — get pushed into a corner by some scalers and left completely alone by others.

"""What one typo does to each scaler.

A height column, normally distributed around 50, into which a single 5000 has
been typed. Every scaler is fitted on the contaminated column, and the question
is what happened to the other 999 rows — the ones that are fine.

`span_of_the_rest` is how much of the output range the middle 98% of the honest
data occupies after scaling. Large is good: it means the real data still uses
the space. Small means the typo pushed everything else into a corner, where the
differences the model needs are below the resolution of its inputs.

Printed as a markdown table, in identifiers only, so one artifact serves both
the English and the Portuguese page.
"""

import numpy as np
from sklearn.preprocessing import (MaxAbsScaler, MinMaxScaler, QuantileTransformer,
                                   RobustScaler, StandardScaler)

N, TYPO = 1000, 5000.0

rng = np.random.default_rng(3)
clean = rng.normal(50, 10, N)
contaminated = clean.copy()
contaminated[0] = TYPO                        # one slipped decimal point

scalers = {
    "StandardScaler": StandardScaler(),
    "MinMaxScaler": MinMaxScaler(),
    "RobustScaler": RobustScaler(),
    "MaxAbsScaler": MaxAbsScaler(),
    "QuantileTransformer": QuantileTransformer(n_quantiles=500, random_state=0),
}


def span(column, scaler, skip_first):
    scaled = scaler.fit_transform(column.reshape(-1, 1)).ravel()
    honest = scaled[1:] if skip_first else scaled
    low, high = np.percentile(honest, [1, 99])
    return high - low


print("| `scaler` | `span_clean` | `span_with_one_typo` | `kept` |")
print("|---|---:|---:|---:|")
for name, scaler in scalers.items():
    before = span(clean, scaler, skip_first=False)
    after = span(contaminated, scaler, skip_first=True)
    print(f"| `{name}` | {before:.3f} | **{after:.3f}** | **{after / before:.1%}** |")

One typo — a single 5000 in a column that lives around 50 — and the kept column says how much of the resolution the other 999 rows still have:

• MinMaxScaler: 1.3%. The typo becomes 1.0 and everything real is crushed into the first hundredth of the range. MaxAbsScaler: 1.7%, for the same reason.
• StandardScaler: 6.4%. Better, because a mean and a standard deviation are moved by one value less violently than a maximum is — but still a fifteen-fold loss of resolution.
• RobustScaler: 100.0%. The median and the quartiles did not move, so nothing happened to the honest rows. This is what "robust" means, stated as a number.
• QuantileTransformer: 99.8%, by construction: it only looks at the ordering, and the typo is simply the last in line.

Choosing

flowchart TD
    A["a numeric column"] --> B{"bounded by nature?<br/><small>pixels, percentages, probabilities</small>"}
    B -->|yes| MM["<b>MinMaxScaler</b><br/><small>the bounds are real, so use them</small>"]
    B -->|no| C{"outliers, or a heavy tail?"}
    C -->|"yes, and they are errors"| R["<b>RobustScaler</b><br/><small>after fixing what you can</small>"]
    C -->|"yes, and they are real"| Q["<b>QuantileTransformer</b><br/><small>or a log — see transforms</small>"]
    C -->|no| S["<b>StandardScaler</b><br/><small>the default, and a good one</small>"]
    A --> D{"sparse matrix?<br/><small>one-hot, tf-idf</small>"}
    D -->|yes| MA["<b>MaxAbsScaler</b><br/><small>the only one that keeps zeros at zero</small>"]

    classDef ok fill:#e6f4ea,stroke:#3fb950,color:#14532d
    classDef q  fill:#eef2f7,stroke:#8b949e,color:#1f2937
    class MM,R,Q,S,MA ok
    class A,B,C,D q

A few decisions that the diagram compresses:

• Sigmoid or tanh in the first layer argues for MinMax to \([0,1]\) or \([-1,1]\): those activations saturate outside a narrow band, and a standardized input with a few values at \(\pm 4\) will sit in the flat region where the gradient is nearly zero. With ReLU it matters much less.
• Sparse data — one-hot blocks, tf-idf — must not be centred, because subtracting a mean turns every structural zero into a non-zero and a sparse matrix into a dense one that may not fit in memory. MaxAbsScaler, or StandardScaler(with_mean=False).
• QuantileTransformer is not free. It forces any distribution into the shape you ask for, which fixes skew and outliers at once — and it throws away all distance information, keeping only the ordering. If the gaps between values mean something, that is a real loss.

scaler.fit_transform(X_train)   # learns mu and sigma here
scaler.transform(X_test)        # and only applies them here

Scaling the target

For regression, the target has units too, and they set the scale of the loss. A target in the millions produces a mean squared error in the trillions and gradients to match.

from sklearn.compose import TransformedTargetRegressor

model = TransformedTargetRegressor(
    regressor=pipeline,
    func=np.log1p, inverse_func=np.expm1,     # or transformer=StandardScaler()
)

The wrapper matters: it inverts the transform before scoring, so your reported error stays in the original units and remains a number somebody can act on. Scaling y by hand and forgetting to invert is a common and silent way to report a meaningless metric.

Where scaling is not enough

Scaling is a linear map — subtract something, divide by something. It moves and stretches a distribution; it cannot change its shape. A right-skewed column is still right-skewed after standardizing, with exactly the same skew coefficient.

When the shape itself is the problem, you need a non-linear transform: that is the next page.