Overview

A model that fits the training data perfectly is not the goal — it is a warning sign. What we actually want is a model that does well on data it has never seen, and those two things pull in opposite directions. Regularization is the name for every technique that deliberately makes the training fit worse in exchange for a better fit on everything else.

Formally, training minimizes the loss we can measure, over the \(N\) examples we happen to have:

but what we care about is the loss over the whole distribution the data came from, which we cannot measure:

The difference \(R(\theta) - \hat{R}(\theta)\) is the generalization gap, and every technique in this chapter is an attempt to keep it small.

What this chapter is about

Every section below answers one question, and they are meant to be read in this order:

flowchart TD
    A["Train and validation error<br/>read together"] --> B{Which failure?}
    B -->|"both high"| C["Underfitting<br/>more capacity, train longer,<br/>regularize less"]
    B -->|"train low,<br/>validation high"| D["Overfitting"]
    D --> E["1. More data<br/>or augmentation"]
    D --> F["2. Constrain the weights<br/>L2 / L1"]
    D --> G["3. Inject noise<br/>dropout, BatchNorm"]
    D --> H["4. Limit the search<br/>early stopping"]
    D --> I["5. Soften the target<br/>label smoothing, mixup"]

Nine live simulators run through the page. None of them needs installing anything — the whole point is that you can move a slider and watch the failure appear and disappear before we write down a single equation about it.

Seeing it happen

Nothing explains overfitting like watching it. Below, 12 noisy points are drawn from a smooth function, and a polynomial of the degree you choose is fitted to them by least squares. The dashed line is the truth we are trying to recover; the crosses are held-out points the fit never saw.

Three experiments worth running, in order:

1. Degree 1, then 3, then 14, with \(\lambda = 0\). Degree 1 cannot bend at all. Degree 3 lands close to the truth. Degree 14 threads every single training point — train error collapses to essentially zero — and lurches violently between them. That is overfitting: not a model that is wrong, but a model that is right about the noise.
2. Stay at degree 14 and raise \(\lambda\). Nothing about the model shrinks; it still has 15 coefficients. But the wild curve calms down and the held-out error drops. Capacity was never the problem — unconstrained capacity was.
3. Stay at degree 14, \(\lambda = 0\), and drag the number of points up. The same model that was useless with 12 points becomes reasonable with 60. More data is regularization too, and the most effective kind.

Bias and variance

The two failures in that simulator have names, and they come from splitting the expected error of a model into pieces. For squared loss, at a point \(x\), averaging over all training sets we might have drawn:

The classic way in is a shooting target. Fix a point \(x\), then imagine training the same model many times, each time on a different draw of the training data, and marking where each of those models lands. The bullseye is the truth \(f(x)\); each dot is one model's prediction.

The two failures are independent — a model can have either, both, or neither — and that is the whole reason the decomposition is worth writing down. Read its three terms as three different complaints:

• Bias — the model is systematically wrong, in the same way, no matter which training set it gets. On the target: the dashed line, from the bullseye to where the dots land on average. The degree-1 fit is biased: it cannot bend, so it misses the curve every time.
• Variance — the model changes wildly depending on which training set it happened to see. On the target: the scatter of the dots around their own average. The degree-14 fit has enormous variance: resample the 12 points and you get a completely different curve.
• Noise — the irreducible part, \(\sigma^2 = 0.032\) in the simulator. On the target: a gust of wind on every shot — no aim removes it. No model beats it, and a model that scores below it on training data is fitting noise by definition.

Model capacity trades the first against the second, which is the classic bias-variance tradeoff:

That is the drawn version. Here is the measured one. For each polynomial degree, the same fit is repeated on 140 different training sets drawn from the same truth, and the error at 41 held-out points is split into the two terms of the equation above — \(\big(\text{average} - \text{truth}\big)^2\) and the scatter around that average:

The U is the white curve, and it is a U for a reason you can now see rather than assert: the two curves underneath it run in opposite directions, so their sum has to have a bottom somewhere. Four things to do with it:

1. Read the two ends. At degree 1 the error is almost entirely bias² — every training set produces roughly the same wrong curve. At degree 11 it is almost entirely variance — every training set produces a different curve. Same total error, opposite diseases, opposite cures.
2. Turn the L2 penalty up. A ghost line appears: the total error without it. The right-hand arm of the U flattens out while the left-hand arm barely moves — the penalty buys a large reduction in variance for a small increase in bias. That trade is the whole business of this chapter.
3. Drag the training points from 12 to 60. The valley widens and the right arm sinks: with more data, high capacity stops being dangerous. Compare with experiment 3 in the first simulator.
4. Drop the noise σ. The optimum moves right — the less noise there is to mistake for signal, the more capacity you can afford.

Penalizing the weights: L1 and L2

The simulator's lesson was that memorizing needs enormous coefficients. So add their size to the loss and let the optimizer weigh the two objectives:

Why L1 gives you zeros and L2 does not

This is easier to see than to prove. Below, the ellipses are the contours of the unregularized loss and the shaded region is what the penalty allows; the penalized solution sits where the smallest ellipse touches that region. Drag the black dot — the unregularized optimum — and watch what each penalty does with it.

The shapes are the whole argument. The L2 region is a circle, which is smooth everywhere, so the point where it first touches an ellipse is almost never on an axis — both coordinates end up small but alive. The L1 region is a diamond, and a diamond is all corners; the corners lie on the axes, and a shrinking ellipse hits a corner for a wide range of directions it could have come from. That is sparsity: not a numerical accident, but a consequence of the shape.

Try dragging \(\hat{w}\) close to one axis and raising \(\lambda\) — the L1 solution snaps onto the axis and stays there, while the L2 solution slides toward the origin without ever arriving.

Dropout: break the co-adaptations

Dropout⁷ attacks a different failure. A network can memorize by building co-adapted groups of neurons: detectors that only work because some other specific neuron is there to correct them. Such a committee is brittle, and it is exactly what fails on new data.

The fix is brutal. At every training step, each unit is deleted independently with probability \(p\):

No unit can rely on any other being present, because any of them may be gone on the next step. Each neuron is forced to be useful on its own, and the network stops putting all its confidence in a few fragile paths.

There is a second reading of the same equation. Every mask is a different thinned subnetwork, and with \(n\) units there are \(2^n\) of them sharing weights. Training with dropout trains that whole ensemble a little at a time, and testing without it approximates averaging their predictions — an ensemble at the price of a single model.

The \(1/(1-p)\) everyone gets wrong

Deleting units at training time lowers the expected value of what the next layer receives. That has to be compensated, and there are two ways to do it:

Every framework in use today does the inverted version, which is why model.eval() simply turns dropout off rather than rescaling anything. Watch the expectation being preserved:

Let it run: the average of the training output converges to the inference output, whatever \(p\) is. That is the entire point of dividing by \(1-p\) — the next layer sees the same expected input in both modes, so the network you train is the network you deploy. Push \(p\) to 0.9 and watch the average take much longer to settle: heavy dropout preserves the mean but adds enormous variance, which is why it slows convergence.

Where dropout is used today

Normalization layers

Batch normalization⁹ standardizes the input of a layer using the statistics of the current mini-batch:

where \(\mu_B\) and \(\sigma_B^2\) are the mean and variance over the batch, and \(\epsilon \approx 10^{-5}\) keeps the root away from zero. The learned \(\gamma\) and \(\beta\) then let the layer undo the normalization if that is what helps — including recovering the identity, so the layer can never cost the network representational power.

Training and inference are different computations

This is the part that trips people up, and it is worth being exact about, because it is also where the regularizing effect comes from.

At training time a sample's normalized value depends on which other samples landed in its batch — so the same image gets a slightly different representation depending on its company. That is noise injected into the forward pass, and it is why batch normalization regularizes at all. At inference the batch is gone and the network must be deterministic, so the stored averages take over.

Note what this does not say: batch normalization is not "turned off" at inference. It always runs — it just switches which statistics it uses. Turning it off would change the scale of every activation and destroy the model.

That last paragraph is easier to believe once you have seen it. Below, one fixed activation \(x^\star = 2.3\) is normalized over and over, each time inside a freshly drawn batch. Nothing about the sample changes — only its company does.

The orange histogram is the same number computed hundreds of times, and it is wide. The green line is what inference computes — one value, always. Two things to try:

1. Drag B from 4 to 128. The histogram tightens around the inference value: bigger batches mean less noise, and so less regularization. This is the reason batch normalization stops helping as a regularizer at very large batch sizes, and why people then add back the noise elsewhere.
2. Go back to B = 4 and look at the mean of the histogram, not just its width. With very small batches the training-time value is not only noisy, it is biased away from what inference will compute — which is why BatchNorm degrades badly at tiny batch sizes and GroupNorm or LayerNorm are used instead.

The backward pass: one sample moves them all

On the way forward, \(\mu\) and \(\sigma^2\) come from the batch. That has a consequence almost everyone gets wrong when implementing it by hand: a sample's gradient does not depend on that sample alone. Nudging \(x_i\) moves the mean and the variance, and with them the \(\hat{x}_j\) of every other sample in the batch.

Writing \(g_i = \gamma\,\partial L/\partial y_i\), with \(m\) for the batch size:

The layer's own two parameters are simpler: each one adds up, over the batch, the gradient that reaches it.

The two sums inside \(\partial L/\partial x_i\) are the two extra paths: the first is the one through the mean, the second the one through the variance. Drop them and the expression collapses to \(g_i/\sigma\), the classic mistake of anyone writing the layer from scratch — and it does not blow up, it just trains worse.

There is a cheap check on the formula: \(\sum_i \partial L/\partial x_i = 0\), always. Shifting the whole batch changes nothing after normalization, so the gradient cannot have a component along that direction. The script below prints that sum and compares the formula with finite differences.

"""A passagem reversa da batch normalization, conferida por diferenças finitas.

O ponto da conta: como μ e σ² dependem do batch inteiro, o gradiente de UMA amostra
carrega termos de TODAS as outras. É por isso que a forma compacta tem duas somas.
"""
import numpy as np

EPS = 1e-4

def forward(x, gamma, beta):
    mu, var = x.mean(), x.var()
    xhat = (x - mu) / np.sqrt(var + EPS)
    return gamma * xhat + beta, xhat, np.sqrt(var + EPS)

def backward(dy, xhat, sigma, gamma):
    m = dy.size
    g = dy * gamma                                    # ∂L/∂x̂
    dx = (m * g - g.sum() - xhat * (g * xhat).sum()) / (m * sigma)
    return dx, (dy * xhat).sum(), dy.sum()            # dx, dgamma, dbeta

x = np.array([1.0, 2.0, 3.0, 4.0])
dy = np.array([0.1, 0.2, -0.1, 0.3])
gamma, beta = 1.0, 0.0

y, xhat, sigma = forward(x, gamma, beta)
dx, dgamma, dbeta = backward(dy, xhat, sigma, gamma)

h, num = 1e-6, np.zeros_like(x)
for i in range(x.size):
    xp, xm = x.copy(), x.copy()
    xp[i] += h; xm[i] -= h
    num[i] = ((dy * forward(xp, gamma, beta)[0]).sum() - (dy * forward(xm, gamma, beta)[0]).sum()) / (2 * h)

np.set_printoptions(precision=4, suppress=True)
print("x       =", x)
print("x̂       =", xhat, " (média 0, desvio 1)")
print("dy      =", dy)
print("dx      =", dx)
print("dgamma  =", round(dgamma, 4), " dbeta =", round(dbeta, 4))
print("soma de dx =", round(dx.sum(), 12), " — sempre zero: deslocar o batch inteiro não muda x̂")
print("erro máximo contra diferenças finitas:", f"{np.abs(num - dx).max():.1e}")

x       = [1. 2. 3. 4.]
x̂       = [-1.3416 -0.4472  0.4472  1.3416]  (média 0, desvio 1)
dy      = [ 0.1  0.2 -0.1  0.3]
dx      = [ 0.0179  0.0805 -0.2147  0.1163]
dgamma  = 0.1342  dbeta = 0.5
soma de dx = 0.0  — sempre zero: deslocar o batch inteiro não muda x̂
erro máximo contra diferenças finitas: 1.7e-10

LayerNorm is the same formula with the sums running over one sample's features instead of over the batch. It is literally the arithmetic the depth panel of chapter 9 runs at every layer.

LayerNorm, and why Transformers use it instead

Batch normalization has a structural weakness: it needs a batch. That breaks with batch size 1, with variable-length sequences, and it makes training and inference behave differently by construction.

LayerNorm¹¹ normalizes over the features of a single sample instead of over the batch:

Each sample is now normalized by its own statistics. There is nothing to average across the batch, so there are no running statistics, training and inference are the same computation, and batch size is irrelevant. That is why every Transformer uses it — and also why LayerNorm does not regularize: no batch, no noise from the batch composition.

The two are easiest to tell apart on a concrete matrix. Below, rows are the samples of a batch and columns are features. Click a cell and the purple outline marks the values its \(\mu\) and \(\sigma\) come from; the box underneath writes out the arithmetic, with \(\gamma = 1\) and \(\beta = 0\). Every button changes one thing and reports what it did to the output of sample 1, the orange row.

Four things to try, each isolating one property:

1. Click a cell and switch between the two. BatchNorm's group is the column — one feature, across the batch. LayerNorm's is the row — one sample, across its features. Same formula, other axis.
2. Redraw the other samples. Under BatchNorm the output of sample 1 moves, though sample 1 did not change: that is the noise the histogram above showed, made of its company. Under LayerNorm it does not move at all.
3. Drag B down to 1. BatchNorm is left with one value per column, which is its own mean: every output becomes 0 and the input is erased. PyTorch refuses to train like that and raises an error. LayerNorm does not notice.
4. Multiply sample 1 by 10, then put feature 3 in thousands. Each normalization is blind to exactly one of these. LayerNorm erases the scale of a sample, so ×10 leaves its output identical. BatchNorm erases the scale of a feature, so the thousands leave every output identical. The other one pays: under BatchNorm the ×10 sample drags the mean and deviation of every column and the rest of the batch moves with it; under LayerNorm feature 3 takes over each sample's \(\sigma\) and the other four features come out almost equal, at about \(-0.5\).

That last pair is the practical rule. When features live on different scales — tabular inputs, the channels of a CNN — per-feature statistics are what you want: BatchNorm. When every coordinate of a vector is the same kind of quantity and samples differ in overall size — the embedding of a token — per-sample statistics are: LayerNorm, which also brings no dependence on the batch.

Stopping early

Train long enough on a model with spare capacity and it will begin to fit the noise. Validation loss records the moment: it falls with the training loss for a while, bottoms out, and then rises while the training loss keeps going down. Early stopping is the decision to keep the model from the bottom of that curve.

The recipe has three parts, and the third is the one people forget:

1. Evaluate on a validation split every epoch (or every \(k\) steps).
2. Patience¹² — how many evaluations without improvement you tolerate before stopping. Too small and you quit on noise; too large and you waste compute past the point of no return.
3. Restore the best checkpoint. Stopping is not enough — the weights in memory when you stop are the worse ones, from patience epochs past the minimum. Keep a copy of the best.

Below, a degree-15 polynomial is fitted to 10 noisy points by plain gradient descent. Nothing is penalized and nothing is dropped; the only thing that varies is when you stop.

The right-hand panel is the point: one training run, two very different models. The green curve — the one you keep — follows the truth. The red one cost 160 times more compute (20 000 epochs against 125), threads every training point, and is worse everywhere in between. Its training loss is genuinely lower. That is the whole trap.

More data, and how to manufacture it

Go back to the first simulator and drag the number of points from 12 to 60 with degree 14 and no penalty. The model that was useless becomes fine. Nothing about the model changed — the constraint came from the data. This is worth stating plainly, because it reorders every priority in this chapter: when overfitting is the problem, more data is the best available answer, and everything else is what you do when you cannot get any.

You often can manufacture some, though, by exploiting invariances you already know about:

train_tf = transforms.Compose([
    transforms.RandomResizedCrop(224, scale=(0.6, 1.0)),
    transforms.RandomHorizontalFlip(),          # a cat, mirrored, is a cat
    transforms.ColorJitter(0.3, 0.3, 0.3),      # so is a cat under warmer light
    transforms.ToTensor(),
])
val_tf = transforms.Compose([                   # never augment validation
    transforms.Resize(256), transforms.CenterCrop(224), transforms.ToTensor(),
])

Label smoothing and mixup are both easier to see than to describe: one changes where the target is, the other changes which examples exist.

On the left, drag \(\alpha\). At \(\alpha = 0.05\) almost every \(\lambda\) lands near 0 or 1, so the mixed points sit right next to real examples — a mild edit. At \(\alpha = 2\) the middle is sampled just as often, and the training set fills up with points that are genuinely half of each class. That is why \(\alpha\) between 0.1 and 0.4 is the usual choice for images: enough to smooth the space between examples, not enough to spend the model on chimeras.

On the right, drag \(\varepsilon\). At \(\varepsilon = 0\) the target is a hard 1 and the optimum is an infinite logit gap — the loss keeps pushing forever, which is exactly what overconfidence is. Any \(\varepsilon > 0\) puts the optimum at a finite, reachable place.

Putting it together

No single technique is the regularizer; real training stacks several, and they are not interchangeable. Sorted by what they actually do:

And now you can stack them yourself. The simulator below trains a real network — 2 → h → h → 1, tanh, full-batch gradient descent — on 40 points from two interleaving moons, 8% of which have the wrong label. Every control is one of the techniques above, and the left panel shows the decision boundary the model actually learned while the right panel shows both losses.

Five experiments, in order. Each one takes about fifteen seconds:

1. Everything off. Train with \(\lambda = 0\), \(p = 0\), \(\sigma = 0\), 24 units. Uncheck show the early-stopped model to see the final weights: the training loss keeps falling while the validation loss turns around and climbs, and the boundary grows a little island around every mislabelled point. Around 95% train against 78% validation.
2. Check the box again. Same run, same compute — but reading the weights from the best validation epoch instead of the last one recovers validation accuracy, from 78% to 81% on this draw of the data. That is early stopping, and it cost nothing.
3. Raise the weight decay. Around \(\lambda \approx 10^{-2}\) the islands dissolve and the boundary becomes one smooth curve; validation catches up with training. Keep going to the top of the slider and watch it collapse into underfitting: the penalty eventually drowns the signal too.
4. Turn weight decay off and raise dropout. \(p = 0.2\) helps. \(p = 0.4\) on a network this small destroys it — there is not enough redundancy left for half the units to disappear. Dropout is a technique for networks with capacity to spare.
5. Turn everything off and drag the training points from 40 to 200. The gap closes on its own, with no regularizer at all. That is the ordering of this chapter in one experiment.

Reading the two curves

Key takeaways

1. The goal is the generalization gap, not the training loss. A training loss of zero is a symptom to investigate, not an achievement.
2. Underfitting and overfitting are diagnosed by two numbers, never one: train error and validation error, read together.
3. Overfitting shows up in the weights — huge coefficients of alternating sign — which is what makes penalizing them work at all.
4. L2 shrinks weights proportionally and never reaches zero; L1 pushes with constant force and produces exact zeros. The difference is the shape of the region each one allows.
5. Dropout breaks co-adaptation and approximates an ensemble of \(2^n\) subnetworks. Modern implementations scale by \(1/(1-p)\) during training, so inference is a plain forward pass.
6. Normalization helps optimization first and regularizes second; batch norm's regularizing noise comes from the batch composition, which is exactly what LayerNorm gives up.
7. Early stopping is not a heuristic — for linear models it is provably close to ridge, with \(\lambda \approx 1/(\eta t)\).
8. More data beats all of it. Augmentation is the second-best thing, because it is more data that you built out of what you already knew.

Additional Resources

Three places to go next, in the order that makes sense:

1. Machine Learning Fundamentals: Bias and Variance — Starmer, J., StatQuest. Ten minutes on the decomposition this chapter opens with, at a gentler pace and with a different set of pictures. Watch it first if the bias-variance section felt abstract:

2. A Recipe for Training Neural Networks — Karpathy, A. The practical counterpart to this page: the order in which to actually do things, why you should overfit a single batch before anything else, and a catalogue of the mistakes that look like overfitting but are bugs.

3. Deep Learning, chapter 7 — Regularization for Deep Learning — Goodfellow, I., Bengio, Y., & Courville, A. The reference treatment, with the derivations this page only gestures at — including the proof that early stopping and L2 coincide for linear models.

References

The works cited through the text, in order of appearance: