9. Deep Learning

Stacking more layers is not a quantitative decision. It is a bet that the thing you are modelling is compositional — that "a face" is made of parts, which are made of edges, which are made of pixels — and that a network which mirrors that structure needs fewer parameters than one that does not.

The MLP chapter already told you a single hidden layer can approximate any continuous function. So depth is not about what a network can represent; it is about how efficiently it represents it, and about whether you can actually train the thing once it is deep.

This chapter is about the bill that comes with the bet. Everything here answers one question: what breaks when you stack layers, and what was invented to fix it?

The signal has to survive the trip

Before any training, before any data, a deep network has to pass a signal from the input to the output and a gradient back. Both cross \(L\) weight matrices, and each crossing multiplies their size by some factor — so what arrives is a product of \(L\) factors, and a product of many numbers does one of two things: it goes to zero, or it goes to infinity. Staying near 1 is the exception you have to engineer.

The panel below builds a network of the depth you choose, runs one forward pass and one backward pass at initialization — no training at all — and follows both trips.

First, what is on the vertical axis. Call \(a\) the value one unit produces, and \(g\) the derivative of the loss with respect to that layer's output. Each layer has 48 units and the batch holds 32 samples, so every layer carries \(48 \times 32\) values of \(a\) and as many of \(g\). The panel boils each layer down to one number, the mean of the squares:

The square is there to measure typical size without positive and negative values cancelling — it is the same quantity the Xavier and He arithmetic of the next section is about. An \(\mathbb{E}[a^2]\) of 100 means activations on the order of 10; of \(10^{-6}\), on the order of 0.001.

Now the two trips:

• On the left, forward. The signal enters at layer 1 and moves right. The curve is each layer's \(\mathbb{E}[a^2]\) divided by layer 1's, so it starts at ① on the left-hand end, worth 1 by construction, and the number that matters is ② at the right-hand end: how many times larger or smaller the signal arrived at the end.
• On the right, backward. The gradient is born at the loss, after the last layer, and walks back to layer 1. So that the curve follows that direction, this plot's axis is reversed: the last layer sits on the left and layer 1 on the right. Both panels then read the same way, left to right, from ① to ② — only the layer numbering runs backwards. The curve is each layer's \(\mathbb{E}[g^2]\) divided by the last layer's, and ② is what is left of the gradient for the early layers.

On both sides 1 means nothing was lost or amplified: it is the dashed line, and the green band around it runs from 0.1× to 10×. The axis is logarithmic and always spans 12 orders of magnitude, so a sloped straight line means exponential growth and the same slope means the same factor in any configuration. It starts at 1 and extends towards wherever the data goes — up when the signal explodes, down when it dies, both ways when it hovers around 1 — with the same range in both panels. When a curve passes those limits it turns dotted, hugging the edge, with an arrow where it left — the value keeps growing, it just no longer fits in the drawing, and the ② label says where it ended up. The thin lines are five networks sampled with different weights; the thick one is their geometric mean.

Six experiments, and they are the chapter in order:

1. ReLU, naive initialization, 30 layers — the state the panel opens in. Weights drawn from \(\mathcal{N}(0,1)\): the signal reaches the last layer \(10^{39}\) times larger, and the gradient reaches the first just as inflated. Both curves leave the chart. In amplitude — the square root — the activations are about \(5 \cdot 10^{19}\) times larger: float32 still holds them, but a squared loss passes its limit of \(3.4 \cdot 10^{38}\) and comes out inf, and in float16, whose limit is 65,504, the forward pass overflows long before the last layer.
2. Switch to Xavier. The explosion is gone, and now both curves fall in a straight line: the signal loses half at every layer and ends between \(10^{-9}\) and \(10^{-10}\) after 30, while the gradient reaches layer 1 just as shrunken. Half of ReLU's outputs are zero, so half the variance goes with them — Xavier was derived for symmetric activations and does not know that.
3. Switch to He. The thick curves stay inside the green band: the variance is preserved on average. Look at the thin lines, though: a single sampled network can end with its signal at 1/200 of where it started, or at 1.5×. The identity \(\mathbb{E}[z^2] = n\sigma^2\,\mathbb{E}[a^2]\) in the next section is an average over every draw of the weights; a 48-unit network is one draw, and its deviation compounds layer by layer. Even so, the gap to the previous two experiments is dozens of orders of magnitude — the factor of two in the initial variance is the difference between a network that trains and one that does not.
4. Turn on residual connections. Now both curves explode, the signal to ~\(10^{13}\) and the gradient to ~\(10^{12}\). Each block adds its output to its input, so the magnitude compounds going forward; going back, the derivative \(1 + \partial f/\partial a\) exceeds 1 at every layer too. The skip connection alone fixes nothing, which is why residual blocks never appear alone.
5. Pick normalization "after the sum". This is the original 2017 Transformer block: \(a \leftarrow \text{Norm}(a + f(a))\). The signal is pinned at 1 and the gradient improves a lot, but does not settle: it reaches layer 1 about 150× larger at 30 layers and over 1,000× at 40. The identity path now crosses a normalizer at every layer, and the damage grows with depth.
6. Switch to "inside the branch". This is pre-LN, \(a \leftarrow a + f(\text{Norm}(a))\), which Transformers have used ever since. The gradient stays at ~50× for 30 layers and ~70× for 40: going from 20 to 40 layers multiplies the gradient by less than 3, against almost 40 for post-LN. The signal now grows, because the residual stream accumulates every block's contribution — but roughly with the square of the depth, not exponentially. And since each block reads a normalized copy of the stream, its size never reaches the input of any block. To see how robust this combination is, set the initialization back to naive: with pre-LN the gradient stays at ~130×; with post-LN, at ~\(10^5\). The Transformers chapter tells what that swap changed in practice.

Initialization: two lines of arithmetic, half the problem

Take one layer, \(z = Wa\) with \(n\) inputs, weights drawn independently with variance \(\sigma^2\) and zero mean. Then

It is worth separating the two variances the formula puts side by side. \(\sigma^2\) is the variance of the weights — your choice, made before any data exists. \(\mathbb{E}[a^2]\) is the second moment of the activations, which depends on the data. The identity above ties them together: choosing the weights' variance is choosing what happens to the size of the activations, layer after layer.

The procedure is this, run once per layer before training:

1. count \(n\), the number of inputs each neuron receives — the fan-in, the number of columns of \(W\);
2. draw each weight separately from a distribution with mean zero and variance \(\sigma^2\): a normal \(\mathcal{N}(0, \sigma^2)\), or a uniform on \([-\sqrt{3\sigma^2},\, +\sqrt{3\sigma^2}]\), which has exactly the same variance;
3. set the biases to zero.

No data enters that calculation — it only looks at the shape of the layer.

For the signal to survive the layer you want the factor \(n\sigma^2\) to be 1, which gives \(\sigma^2 = 1/n\) — Xavier (Glorot) initialization¹. With ReLU, half the outputs are set to zero, so the activation itself halves the second moment; compensating for that requires twice the variance, \(\sigma^2 = 2/n\) — He initialization².

The factor of one layer and what moves it

The number the identity above produces deserves a name. Every layer multiplies the second moment by

where \(\rho\) is the fraction of the second moment the activation lets through: exactly \(\tfrac{1}{2}\) for ReLU, which zeroes half its outputs, and at most 1 for tanh, less the more it saturates.

That factor is the subject of the whole chapter, and it is drawn in the panel: the slope of the line is the \(\log_{10}\) of the factor. That is why a straight line on a logarithmic plot means exponential — and it is the number the panel reports in its footer, measured, for every configuration:

In the three ReLU rows the measured factor sits the same 5% below the prediction going forward, and 2% coming back. The prediction is an average over draws of the weights; the panel takes a geometric mean over five networks, and the geometric mean of a quantity that fluctuates sits below its average — the thin lines are that fluctuation. The tanh row is further off for another reason: its \(\rho\) is below 1.

Five things move it, and the first four are controls on the panel — the width is fixed at 48:

• The initialization, linearly: that is \(\sigma^2\), and it is the only term that comes for free.
• The activation, through \(\rho\). Switch to tanh with He and the measured factor is 0.98, not the predicted 2: the larger weights saturate the tanh, \(\rho\) collapses, and the simple arithmetic stops holding.
• The width \(n\), linearly too — which is exactly why Xavier and He carry \(n\) in the denominator, to cancel it. With the naive \(\sigma^2 = 1\) it does not cancel, and the 24 in the first row is just \(48 \cdot \tfrac{1}{2}\).
• The architecture. The residual shortcut adds the identity path to the branch's and takes the factor to about 2.8 forward and 2.6 backward (experiment 4). Normalization after the sum pins the forward factor at exactly 1.00, but leaves the backward one at 1.19.
• Training, which changes the weights and lets the factor drift. Initialization picks the starting point; the shortcut and normalization are what hold it.

Depth is what does not appear on that list: in a plain stack it does not change the factor, it raises it to the \(L\). A factor of 0.95 is harmless at 5 layers and worth \(10^{-7}\) at 300.

x = next(iter(loader))[0]
for name, layer in model.named_children():
    x = layer(x)
    print(f"{name:20s} std={x.std().item():.4f}  mean={x.mean().item():+.4f}")

Residual connections: an identity path for the gradient

The other half of the fix changes the architecture. Instead of asking a block to produce the next representation, ask it to produce the change to the current one³:

Differentiate, and the reason it works is visible:

The 1 is a path along which the gradient reaches the early layers unattenuated, whatever the block does. In a plain network the gradient is a product of \(L\) Jacobians⁴, and a product does only two things: with a factor below 1 per layer it dies exponentially — that is the panel's Xavier, whose gradient is multiplied by 0.49 per layer; with a factor above 1 it explodes, like the naive one, whose predicted factor is \(n\sigma^2/2 = 24\) and whose measured factor on the way back is 23.5. In a residual network there is always a route that multiplies by one.

That 1, though, is a floor and not a ceiling: it keeps the gradient from vanishing, not from growing. Experiment 4 in the panel is exactly that — the skip connection alone multiplies by about 2.8 per block going forward and 2.6 coming back.

The diagram shows both passes through the same block. Going forward, the skip copies \(a_l\) and adds it to the output of the block's layers. Going back, the gradient \(g\) arriving from the loss is copied onto both paths: through the layers it is multiplied by \(\partial f/\partial a_l\), which can be tiny; through the skip it is multiplied by 1. The two parts add up again at the block's input.

There is a second reading, and for teaching it is the better one: a residual block starts life as the identity function if \(f\) starts near zero. Adding a block to a trained network therefore costs nothing — the network you had is still in there, and the block only has to learn what to add. Depth stops being a risk and becomes an option, which is how a 152-layer network beat a 19-layer one³ instead of collapsing.

Normalization, and where it lives in this course

The third piece of the answer is normalization, and it is treated with its own simulator in the regularization chapter: what BatchNorm computes, why training and inference are different computations, and why LayerNorm — not BatchNorm — is what Transformers use.

In a deep network it is applied layer by layer: there is one normalization inside every block, repeated across the \(L\) layers — not a single standardization of the data at the network's input. Two questions tend to get mixed up, and the diagram separates them. Where it goes in the block is what tells post-LN from pre-LN in experiments 5 and 6. What it takes the mean and deviation over is what tells LayerNorm, which the panel applies across the 48 units of each sample, from BatchNorm, which uses the whole batch for each unit.

Two facts from there matter for depth, and they are worth carrying:

• Normalization re-centres and re-scales the signal at every layer, which is the direct fix for the explosion above and — together with the skip connection — the reason very deep stacks are trainable at all. On its own it is not enough: pick "after the sum" with residual connections off, and the signal holds at 1 while the gradient reaches layer 1 hundreds of thousands of times larger. The backward pass divides the gradient by the same deviation the forward pass divided the signal by, but that deviation is taken after the mean is removed, and a ReLU's output is all positive: its deviation is smaller than its size, and each layer amplifies the gradient by about 1.5. Try the three initializations there — they give exactly the same curve, because the normalizer erases the scale of the weights.
• Its regularizing effect is a side effect of batch composition. That is why LayerNorm, which uses no batch statistics, normalizes just as well and has no such effect.

What depth is made of

These pieces combine into the architectures that fill the rest of the course:

Key takeaways

1. Depth is a bet that the problem is compositional. A single wide layer can already approximate anything; depth is about doing it efficiently.
2. What breaks first is not accuracy, it is arithmetic: signal and gradient are products of \(L\) terms, and products either vanish or explode.
3. This is visible at initialization, before any training. Check the per-layer activation statistics before blaming the optimizer.
4. Xavier (\(1/n\)) for symmetric activations, He (\(2/n\)) for ReLU. The factor of two is not a detail: at 30 layers it is the difference between \(10^{-9}\) and something of order \(1\) — on average, because a finite network is one draw and strays from it.
5. Residual connections give the gradient an identity path — \(1 + \partial f/\partial a\) — and let a block start as the identity, so depth becomes optional rather than risky.
6. A skip connection alone grows both signal and gradient; it is always paired with normalization, and where that goes matters. After the sum (post-LN) the gradient gets exponentially worse with depth; inside the branch (pre-LN, today's default) it barely does.
7. Architectures are priors — most often expressed as weight constraints — not extra capacity.

Additional resources

• Dive into Deep Learning — Zhang, A., et al. (2020). An open book, with the code next to every derivation.
• Deep Learning — Goodfellow, I., Bengio, Y., & Courville, A. (2016). The canonical reference for the field.
• The Little Book of Deep Learning — Fleuret, F. A hundred pages, for the essentials in one sitting.
• A Survey of Deep Learning: From Activations to Transformers — Schneider, J., & Vlachos, M. (2024). A map of the ideas that stuck, with the lineage of each.

Where each layer is treated

This chapter is about what happens when you stack a lot of anything: what survives the trip, what disappears, and what has to be built for depth to buy you something. The arithmetic of each layer type — forward, backward and a worked example — lives in the chapter that teaches that layer, each with its own simulator: