10. Convolutional Networks

A convolutional layer is a dense layer that has been forbidden two things: it may not connect a pixel to a distant pixel, and it may not use a different weight in a different place. Nothing is added. Capacity is removed, deliberately.

That is the whole idea, and it is worth stating as a claim about the world rather than about code: in an image, what matters is local, and it means the same thing wherever it appears. An edge in the top-left corner is the same event as an edge in the bottom-right. A dense layer does not know that and has to learn it — separately, at every position, from examples. A convolution is handed it for free.

The deep-learning chapter called architectures priors expressed as weight constraints. This chapter is that sentence worked out in full for one case, and then the honest 2026 accounting of where the prior still pays.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. The four simulators of this page run inside the slides. Open the slides in a new tab

The constraint, priced

Below is one 3×3 kernel sliding over a 9×9 input. Press play and watch what is actually happening: the same nine numbers are read at every position, and each output cell is one dot product.

The line under the panel is the argument. Nine weights produce the entire feature map; the dense layer that maps the same inputs to the same outputs needs thousands, and — this is the part that matters more than the parameter count — it would have to rediscover the edge detector at each position from data it does not have.

Two knobs change the shape of the output and both come up constantly in practice:

• Stride \(s\) — how far the window jumps. \(s = 2\) halves the resolution and is the cheapest form of downsampling; it is also how the receptive field gets to grow geometrically (next section).
• Padding \(p\) — invented zeros around the border, so the output keeps the input's size. padding = (k-1)/2 with odd \(k\) is "same" padding, and is the default you should assume.

x = torch.zeros(1, 3, 224, 224)
for name, layer in model.features.named_children():
    x = layer(x); print(f"{name:16s} {tuple(x.shape)}")

From one kernel to a layer: channels

The panel slides one kernel over one grayscale map. A real layer does that many times at once. Its input has \(C_{in}\) channels — three for an RGB image, sixty-four for the output of a previous layer — so each kernel is a \(C_{in} \times k \times k\) block that reads every input channel at the window and adds them all into one number. One kernel gives one feature map; a layer with \(C_{out}\) kernels gives \(C_{out}\) maps, stacked as the channels the next layer reads.

There is no \(H\) and no \(W\) in that count. Conv2d(3, 64, 3) has \(64\,(3 \cdot 3^2 + 1) = 1\,792\) parameters whether the image is 32×32 or 4K. The dense layer that maps a 3×32×32 image to the same 64×32×32 outputs has \(65\,536 \times 3\,073 \approx 201\) million.

Equivariance is what you get; invariance is what you wanted

Weight sharing gives a precise, provable property: shift the input, and the feature map shifts by the same amount. That is translation equivariance, and it is exactly right for a feature detector.

It is not what a classifier needs. A classifier needs invariance — "cat" regardless of where the cat is. Equivariance becomes invariance only when you collapse the spatial axes or when you show the network the transformation, and there are three honest ways to do it:

The receptive field is the real budget

A unit deep in the stack does not see the image. It sees whatever fed it, recursively. That set is its receptive field, and its growth is the constraint that shaped every convolutional architecture ever designed.

Move the sliders and three facts fall out:

1. With stride 1, reach grows linearly. Each 3×3 layer adds 2. Reaching across a 224-pixel image takes over a hundred layers — which is why no architecture is built that way.
2. Stride is what makes it grow geometrically. Every stride-2 layer doubles the step of every layer above it. This — not compute — is the reason CNNs downsample: it is the only affordable route to global context.
3. Dilation buys reach without weights or downsampling, by reading with gaps. It is why dilated convolutions took over semantic segmentation, where you need context and full resolution⁶.

The arithmetic, in one place

For input \(X \in \mathbb{R}^{B \times C_{in} \times H \times W}\) and kernel \(W \in \mathbb{R}^{C_{out} \times C_{in} \times k \times k}\):

The backward pass is the same operation three times, which is the reason convolutions are fast on hardware built for one kernel:

The rotation is not a trick: differentiating \(Y = W \star X\) with respect to \(X\) swaps the roles of the two arguments, and swapping them reverses the index order. With stride \(s > 1\) the recipe is the same after one extra step: insert \(s - 1\) zeros between the entries of the output gradient, so that each of them lands back where its window started. The im2col view, which turns the whole thing into one matrix multiply, is in CS231n¹¹.

The same arithmetic, with numbers

A 3×3 input, a 2×2 kernel and a bias of 1, at stride 1 with no padding:

Each output position is one window multiplied term by term and summed. The first is \(1{\cdot}1 + 2{\cdot}0 + 4{\cdot}(-1) + 5{\cdot}1 + 1 = 3\), and the four together give

Now suppose the loss hands back \(\partial L/\partial Y = \begin{bmatrix} 0.5 & -0.5 \\ 1 & 0 \end{bmatrix}\). The three gradients from the table, on those numbers:

• Bias — sum them all: \(0.5 - 0.5 + 1 + 0 = 1\).
• Kernel — cross-correlate the input with the gradient. The first entry is \(0.5{\cdot}1 - 0.5{\cdot}2 + 1{\cdot}4 + 0{\cdot}5 = 3.5\), and the whole matrix comes out as \(\begin{bmatrix} 3.5 & 4.5 \\ 6.5 & 7.5 \end{bmatrix}\).
• Input — a full convolution of the gradient with the kernel: \(\begin{bmatrix} 0.5 & -0.5 & 0 \\ 0.5 & 1 & -0.5 \\ -1 & 1 & 0 \end{bmatrix}\).

Look at the shapes: the input gradient is 3×3, the size of \(X\), while what arrived was 2×2. Every pixel took part in as many windows as the kernel could reach it in — the corner in one, the centre in four — and the full convolution is exactly the sum over all of them. That is why it needs \(k-1\) of padding on each side, not \(k-1\) in total.

"""A passagem reversa de uma convolução, com números, conferida por diferenças finitas.

As três derivadas são convoluções — é isso que a página afirma e é isso que o print mostra.
"""
import numpy as np
from scipy.signal import correlate2d, convolve2d

def forward(X, K, b):
    return correlate2d(X, K, mode="valid") + b        # correlação cruzada: é o que as bibliotecas chamam de conv

def backward(dY, X, K):
    dX = convolve2d(dY, K, mode="full")               # full: dY cresce k-1 de CADA lado
    dK = correlate2d(X, dY, mode="valid")
    db = dY.sum()
    return dX, dK, db

X = np.array([[1., 2., 3.], [4., 5., 6.], [7., 8., 9.]])
K = np.array([[1., 0.], [-1., 1.]])
b = 1.0
dY = np.array([[0.5, -0.5], [1.0, 0.0]])              # gradiente que chega de cima

Y = forward(X, K, b)
dX, dK, db = backward(dY, X, K)

# Conferência: L = sum(dY * Y) tem gradiente exatamente dY, então dL/dX deve bater com dX.
eps, num = 1e-6, np.zeros_like(X)
for i in range(X.shape[0]):
    for j in range(X.shape[1]):
        Xp, Xm = X.copy(), X.copy()
        Xp[i, j] += eps; Xm[i, j] -= eps
        num[i, j] = ((dY * forward(Xp, K, b)).sum() - (dY * forward(Xm, K, b)).sum()) / (2 * eps)

np.set_printoptions(precision=2, suppress=True)
print("Y  =\n", Y)
print("dY =\n", dY)
print("dX =\n", dX)
print("dK =\n", dK)
print("db =", db)
print("erro máximo contra diferenças finitas:", f"{np.abs(num - dX).max():.1e}")

Y  =
 [[3. 4.]
 [6. 7.]]
dY =
 [[ 0.5 -0.5]
 [ 1.   0. ]]
dX =
 [[ 0.5 -0.5  0. ]
 [ 0.5  1.  -0.5]
 [-1.   1.   0. ]]
dK =
 [[3.5 4.5]
 [6.5 7.5]]
db = 1.0
erro máximo contra diferenças finitas: 5.1e-10

The last line is a finite-difference check. Without one, a padding mistake like the one above goes unnoticed, because the output still comes out in a plausible shape.

The same computation, as the lab writes it: the forward of conv2d_loop, then the three gradients of section 6, line by line. The default case is the example above. Click a cell of a result to see the sum that produced it, then turn on padding, set the stride to 2 or pick another case and watch which lines change:

Pooling: what it throws away, and where the gradient goes

A pooling layer has no weights at all. It summarizes each window into a number, and the choice of summary decides who receives gradient:

On the way back, max splits nothing. The whole window gets zero except the position that won, which gets the entire gradient:

Average swaps that indicator for \(1/k^2\), the same sum with the credit split evenly. The panel shows both, with a gradient of 1 at every output position, so that the number you see coming back is literally how many windows that cell won.

In the default setting — a 2×2 window at stride 2 — 27 of the 36 input cells get nothing. They influenced no output, so the loss does not move when they change a little. Three consequences worth carrying:

• Max pooling is a router, not a filter. It picks one path per window, and the gradient only travels down that one. Switch to average and every cell receives \(1/k^2\) — steadier, less selective.
• The route changes during training. Today's winner need not be tomorrow's, because it depends on the values rather than the indices. That is why max pooling is not the fixed operation it looks like.
• With a stride smaller than the window, windows overlap and a cell can accumulate gradient from several of them. Drag the stride to 1 and watch the number climb.

"""Max pooling: a ida guarda o índice do vencedor, a volta devolve o gradiente só para ele."""
import numpy as np

def forward(X, k=2, s=2):
    H, W = X.shape
    Y = np.zeros((H // s, W // s))
    arg = np.zeros_like(Y, dtype=int)                 # índice achatado do vencedor de cada janela
    for i in range(0, H - k + 1, s):
        for j in range(0, W - k + 1, s):
            win = X[i:i + k, j:j + k]
            arg[i // s, j // s] = np.argmax(win)       # empate: argmax escolhe o primeiro
            Y[i // s, j // s] = win.flat[arg[i // s, j // s]]
    return Y, arg

def backward(dY, arg, shape, k=2, s=2):
    dX = np.zeros(shape)
    for i in range(dY.shape[0]):
        for j in range(dY.shape[1]):
            di, dj = divmod(arg[i, j], k)
            dX[i * s + di, j * s + dj] = dY[i, j]      # o resto da janela fica em zero
    return dX

X = np.arange(1., 17.).reshape(4, 4)
dY = np.array([[0.5, -0.5], [1.0, 0.0]])

Y, arg = forward(X)
dX = backward(dY, arg, X.shape)

np.set_printoptions(precision=2, suppress=True)
print("X  =\n", X)
print("Y  =\n", Y)
print("dY =\n", dY)
print("dX =\n", dX)
print("gradiente que chegou:", dY.sum(), "· gradiente distribuído:", dX.sum(),
      "· posições que receberam algo:", int((dX != 0).sum()), "de", X.size)

X  =
 [[ 1.  2.  3.  4.]
 [ 5.  6.  7.  8.]
 [ 9. 10. 11. 12.]
 [13. 14. 15. 16.]]
Y  =
 [[ 6.  8.]
 [14. 16.]]
dY =
 [[ 0.5 -0.5]
 [ 1.   0. ]]
dX =
 [[ 0.   0.   0.   0. ]
 [ 0.   0.5  0.  -0.5]
 [ 0.   0.   0.   0. ]
 [ 0.   1.   0.   0. ]]
gradiente que chegou: 1.0 · gradiente distribuído: 1.0 · posições que receberam algo: 3 de 16

Flatten: the bridge to the dense head

Between the last convolution and a dense layer, the \((B, C, H, W)\) tensor has to be unrolled into a \((B,\; C \cdot H \cdot W)\) vector. That is flatten: no parameters, no arithmetic, and a backward pass that only puts the original shape back. What it does have is a hidden cost — it is what turns the whole map into the input of one enormous dense matrix, and that is where most of AlexNet's and VGG's parameters lived. Replacing it with global average pooling, in the table above, removed that cost outright.

A whole CNN, layer by layer

Everything above, assembled. This is the network the lab trains on Fashion-MNIST (28×28 grayscale), with the shape, the parameter count and the receptive field after every layer:

Three patterns to read off it, and they hold in every CNN since VGG:

• Resolution goes down, channels go up. Each pooling halves \(H\) and \(W\), and the next convolution doubles the channels. The tensor gets smaller while what it describes gets richer: fewer positions, more questions asked about each.
• The parameters live in the late layers. The first convolution costs 192; the third costs 18 624, because its cost grows with \(C_{in} \cdot C_{out}\), not with the image.
• The head is cheap because of global average pooling. With Flatten instead, the last layer would read \(64 \cdot 7 \cdot 7 = 3\,136\) numbers and cost 31 370 parameters on its own — more than the whole network above — and it would tie each of them to a position. An MLP that reaches similar accuracy, 784 → 256 → 128 → 10, has 235 146.

How the architecture got its shape

Two decades of vision research — from LeNet-5¹ and AlexNet² to ConvNeXt — left a handful of moves that are still in every model, and a larger pile that is not. It is worth knowing which is which.

ConvNeXt: the control experiment

It is tempting to conclude from 2020–2021 that attention beat convolution. ConvNeXt⁸ is the experiment that tests it properly: take a ResNet-50, and change only the things that are not attention — the training recipe (AdamW, 300 epochs, heavy augmentation), the stage ratios, a 7×7 depthwise kernel, LayerNorm instead of BatchNorm, GELU instead of ReLU, fewer normalizations per block. No attention anywhere.

The result matches Swin Transformer at every scale. The honest reading is not "convolutions win" — it is that a large part of the reported ViT advantage was the training recipe, not the architecture, and that the two families converged on the same block design from opposite directions. Where the architectures still genuinely differ is data scale: see chapter 13.

Where convolutions actually live in 2026

They lost the headline task and kept almost everything else. This matters for your project choices:

Lab: the constraint, measured

Every claim on this page is measured in a Colab notebook on Fashion-MNIST (70 000 grayscale clothing images, 28 × 28). It runs in the browser, on CPU; there is nothing to install. Nine sections, each ending in a checkpoint with a check value:

1. a convolution written by hand and checked against PyTorch;
2. the shape error of the first Linear, read instead of guessed;
3. the parameter bill of locality and weight sharing, and of the separable convolution;
4. equivariance and invariance, measured, and why the border breaks the first;
5. the receptive field read off a gradient, theoretical and effective;
6. the backward pass of a convolution and of max pooling, against autograd;
7. an MLP against two CNNs that differ only in the head, on test images shifted until the difference shows, and then rotated;
8. what the first layer learned;
9. a CNN of your own, under a 30 000-parameter budget, scored on the shifted test set.

Open the lab in Colab — save a copy to your Drive (File → Save a copy in Drive) to run and edit it.

Key takeaways

1. A convolution is a dense layer with locality and weight sharing imposed. It is capacity removed, not added, and both constraints are claims about images. Its parameter count, \(C_{out}(C_{in}k^2 + 1)\), does not depend on the image size.
2. Weight sharing gives translation equivariance. Invariance comes from global pooling and from augmentation — and nothing gives rotation or scale invariance for free.
3. The receptive field is the design budget. Stride is what makes it grow geometrically; dilation buys reach without downsampling.
4. Two 3×3 layers beat one 5×5 — fewer weights, more nonlinearity. This is why 3×3 is the default.
5. The backward pass is three convolutions; used forwards, the input-gradient one is transposed convolution, and it checkerboards. Prefer upsample + convolve.
6. Of the classic architecture, what survived is 3×3 stacks, residuals, normalization, global average pooling, depthwise separable and 1×1 convolutions. The big FC heads, LRN and large stem kernels did not.
7. ConvNeXt shows much of the ViT advantage was the training recipe. The families converged.
8. Convolutions lost the ImageNet backbone race and kept the latent autoencoders, medical segmentation, on-device vision and the patch stem — which is most of the deployed pixels in the world.