10. Convolutional

A convolutional layer is a dense layer that has been forbidden two things: it may not connect a pixel to a distant pixel, and it may not use a different weight in a different place. Nothing is added. Capacity is removed, deliberately.

That is the whole idea, and it is worth stating as a claim about the world rather than about code: in an image, what matters is local, and it means the same thing wherever it appears. An edge in the top-left corner is the same event as an edge in the bottom-right. A dense layer does not know that and has to learn it — separately, at every position, from examples. A convolution is handed it for free.

The deep-learning chapter called architectures priors expressed as weight constraints. This chapter is that sentence worked out in full for one case, and then the honest 2026 accounting of where the prior still pays.

The constraint, priced

Below is one 3×3 kernel sliding over a 9×9 input. Press play and watch what is actually happening: the same nine numbers are read at every position, and each output cell is one dot product.

The line under the panel is the argument. Nine weights produce the entire feature map; the dense layer that maps the same inputs to the same outputs needs thousands, and — this is the part that matters more than the parameter count — it would have to rediscover the edge detector at each position from data it does not have.

Two knobs change the shape of the output and both come up constantly in practice:

• Stride \(s\) — how far the window jumps. \(s = 2\) halves the resolution and is the cheapest form of downsampling; it is also how the receptive field gets to grow geometrically (next section).
• Padding \(p\) — invented zeros around the border, so the output keeps the input's size. padding = (k-1)/2 with odd \(k\) is "same" padding, and is the default you should assume.

x = torch.zeros(1, 3, 224, 224)
for name, layer in model.features.named_children():
    x = layer(x); print(f"{name:16s} {tuple(x.shape)}")

Equivariance is what you get; invariance is what you wanted

Weight sharing gives a precise, provable property: shift the input, and the feature map shifts by the same amount. That is translation equivariance, and it is exactly right for a feature detector.

It is not what a classifier needs. A classifier needs invariance — "cat" regardless of where the cat is. Equivariance becomes invariance only when you collapse the spatial axes, and there are two honest ways to do it:

The receptive field is the real budget

A unit deep in the stack does not see the image. It sees whatever fed it, recursively. That set is its receptive field, and its growth is the constraint that shaped every convolutional architecture ever designed.

Move the sliders and three facts fall out:

1. With stride 1, reach grows linearly. Each 3×3 layer adds 2. Reaching across a 224-pixel image takes over a hundred layers — which is why no architecture is built that way.
2. Stride is what makes it grow geometrically. Every stride-2 layer doubles the step of every layer above it. This — not compute — is the reason CNNs downsample: it is the only affordable route to global context.
3. Dilation buys reach without weights or downsampling, by reading with gaps. It is why dilated convolutions took over semantic segmentation, where you need context and full resolution⁶.

The arithmetic, in one place

For input \(X \in \mathbb{R}^{B \times C_{in} \times H \times W}\) and kernel \(W \in \mathbb{R}^{C_{out} \times C_{in} \times k \times k}\):

The backward pass is the same operation three times, which is the reason convolutions are fast on hardware built for one kernel:

The rotation is not a trick: differentiating \(Y = W \star X\) with respect to \(X\) swaps the roles of the two arguments, and swapping them reverses the index order. The im2col view, which turns the whole thing into one matrix multiply, is in CS231n¹¹.

The same arithmetic, with numbers

A 3×3 input, a 2×2 kernel and a bias of 1, at stride 1 with no padding:

Each output position is one window multiplied term by term and summed. The first is \(1{\cdot}1 + 2{\cdot}0 + 4{\cdot}(-1) + 5{\cdot}1 + 1 = 3\), and the four together give

Now suppose the loss hands back \(\partial L/\partial Y = \begin{bmatrix} 0.5 & -0.5 \\ 1 & 0 \end{bmatrix}\). The three gradients from the table, on those numbers:

• Bias — sum them all: \(0.5 - 0.5 + 1 + 0 = 1\).
• Kernel — cross-correlate the input with the gradient. The first entry is \(0.5{\cdot}1 - 0.5{\cdot}2 + 1{\cdot}4 + 0{\cdot}5 = 3.5\), and the whole matrix comes out as \(\begin{bmatrix} 3.5 & 4.5 \\ 6.5 & 7.5 \end{bmatrix}\).
• Input — a full convolution of the gradient with the kernel: \(\begin{bmatrix} 0.5 & -0.5 & 0 \\ 0.5 & 1 & -0.5 \\ -1 & 1 & 0 \end{bmatrix}\).

Look at the shapes: the input gradient is 3×3, the size of \(X\), while what arrived was 2×2. Every pixel took part in as many windows as the kernel could reach it in — the corner in one, the centre in four — and the full convolution is exactly the sum over all of them. That is why it needs \(k-1\) of padding on each side, not \(k-1\) in total.

"""A passagem reversa de uma convolução, com números, conferida por diferenças finitas.

As três derivadas são convoluções — é isso que a página afirma e é isso que o print mostra.
"""
import numpy as np
from scipy.signal import correlate2d, convolve2d

def forward(X, K, b):
    return correlate2d(X, K, mode="valid") + b        # correlação cruzada: é o que as bibliotecas chamam de conv

def backward(dY, X, K):
    dX = convolve2d(dY, K, mode="full")               # full: dY cresce k-1 de CADA lado
    dK = correlate2d(X, dY, mode="valid")
    db = dY.sum()
    return dX, dK, db

X = np.array([[1., 2., 3.], [4., 5., 6.], [7., 8., 9.]])
K = np.array([[1., 0.], [-1., 1.]])
b = 1.0
dY = np.array([[0.5, -0.5], [1.0, 0.0]])              # gradiente que chega de cima

Y = forward(X, K, b)
dX, dK, db = backward(dY, X, K)

# Conferência: L = sum(dY * Y) tem gradiente exatamente dY, então dL/dX deve bater com dX.
eps, num = 1e-6, np.zeros_like(X)
for i in range(X.shape[0]):
    for j in range(X.shape[1]):
        Xp, Xm = X.copy(), X.copy()
        Xp[i, j] += eps; Xm[i, j] -= eps
        num[i, j] = ((dY * forward(Xp, K, b)).sum() - (dY * forward(Xm, K, b)).sum()) / (2 * eps)

np.set_printoptions(precision=2, suppress=True)
print("Y  =\n", Y)
print("dY =\n", dY)
print("dX =\n", dX)
print("dK =\n", dK)
print("db =", db)
print("erro máximo contra diferenças finitas:", f"{np.abs(num - dX).max():.1e}")

Y  =
 [[3. 4.]
 [6. 7.]]
dY =
 [[ 0.5 -0.5]
 [ 1.   0. ]]
dX =
 [[ 0.5 -0.5  0. ]
 [ 0.5  1.  -0.5]
 [-1.   1.   0. ]]
dK =
 [[3.5 4.5]
 [6.5 7.5]]
db = 1.0
erro máximo contra diferenças finitas: 5.1e-10

The last line is a finite-difference check. Without one, a padding mistake like the one above goes unnoticed, because the output still comes out in a plausible shape.

Pooling: what it throws away, and where the gradient goes

A pooling layer has not one weight. It summarizes each window into a number, and the choice of summary decides who receives gradient:

On the way back, max splits nothing. The whole window gets zero except the position that won, which gets the entire gradient:

Average swaps that indicator for \(1/k^2\), the same sum with the credit split evenly. The panel shows both, with a gradient of 1 at every output position, so that the number you see coming back is literally how many windows that cell won.

In the default setting — a 2×2 window at stride 2 — 27 of the 36 input cells get nothing. They influenced no output, so the loss does not move when they change a little. Three consequences worth carrying:

• Max pooling is a router, not a filter. It picks one path per window, and the gradient only travels down that one. Switch to average and every cell receives \(1/k^2\) — steadier, less selective.
• The route changes during training. Today's winner need not be tomorrow's, because it depends on the values rather than the indices. That is why max pooling is not the fixed operation it looks like.
• With a stride smaller than the window, windows overlap and a cell can accumulate gradient from several of them. Drag the stride to 1 and watch the number climb.

"""Max pooling: a ida guarda o índice do vencedor, a volta devolve o gradiente só para ele."""
import numpy as np

def forward(X, k=2, s=2):
    H, W = X.shape
    Y = np.zeros((H // s, W // s))
    arg = np.zeros_like(Y, dtype=int)                 # índice achatado do vencedor de cada janela
    for i in range(0, H - k + 1, s):
        for j in range(0, W - k + 1, s):
            win = X[i:i + k, j:j + k]
            arg[i // s, j // s] = np.argmax(win)       # empate: argmax escolhe o primeiro
            Y[i // s, j // s] = win.flat[arg[i // s, j // s]]
    return Y, arg

def backward(dY, arg, shape, k=2, s=2):
    dX = np.zeros(shape)
    for i in range(dY.shape[0]):
        for j in range(dY.shape[1]):
            di, dj = divmod(arg[i, j], k)
            dX[i * s + di, j * s + dj] = dY[i, j]      # o resto da janela fica em zero
    return dX

X = np.arange(1., 17.).reshape(4, 4)
dY = np.array([[0.5, -0.5], [1.0, 0.0]])

Y, arg = forward(X)
dX = backward(dY, arg, X.shape)

np.set_printoptions(precision=2, suppress=True)
print("X  =\n", X)
print("Y  =\n", Y)
print("dY =\n", dY)
print("dX =\n", dX)
print("gradiente que chegou:", dY.sum(), "· gradiente distribuído:", dX.sum(),
      "· posições que receberam algo:", int((dX != 0).sum()), "de", X.size)

X  =
 [[ 1.  2.  3.  4.]
 [ 5.  6.  7.  8.]
 [ 9. 10. 11. 12.]
 [13. 14. 15. 16.]]
Y  =
 [[ 6.  8.]
 [14. 16.]]
dY =
 [[ 0.5 -0.5]
 [ 1.   0. ]]
dX =
 [[ 0.   0.   0.   0. ]
 [ 0.   0.5  0.  -0.5]
 [ 0.   0.   0.   0. ]
 [ 0.   1.   0.   0. ]]
gradiente que chegou: 1.0 · gradiente distribuído: 1.0 · posições que receberam algo: 3 de 16

Flatten: the bridge to the dense head

Between the last convolution and a dense layer, the \((B, C, H, W)\) tensor has to be unrolled into a \((B,\; C \cdot H \cdot W)\) vector. That is flatten: no parameters, no arithmetic, and a backward pass that only puts the original shape back. What it does have is a hidden cost — it is what turns the whole map into the input of one enormous dense matrix, and that is where most of AlexNet's and VGG's parameters lived. Replacing it with global average pooling, in the table above, removed that cost outright.

How the architecture got its shape

Fifteen years of vision research — from LeNet-5¹ through AlexNet² — left a handful of moves that are still in every model, and a larger pile that is not. It is worth knowing which is which.

ConvNeXt: the control experiment

It is tempting to conclude from 2020–2021 that attention beat convolution. ConvNeXt⁸ is the experiment that tests it properly: take a ResNet-50, and change only the things that are not attention — the training recipe (AdamW, 300 epochs, heavy augmentation), the stage ratios, a 7×7 depthwise kernel, LayerNorm instead of BatchNorm, GELU instead of ReLU, fewer normalizations per block. No attention anywhere.

The result matches Swin Transformer at every scale. The honest reading is not "convolutions win" — it is that a large part of the reported ViT advantage was the training recipe, not the architecture, and that the two families converged on the same block design from opposite directions. Where the architectures still genuinely differ is data scale: see chapter 13.

Where convolutions actually live in 2026

They lost the headline task and kept almost everything else. This matters for your project choices:

Key takeaways

1. A convolution is a dense layer with locality and weight sharing imposed. It is capacity removed, not added, and both constraints are claims about images.
2. Weight sharing gives translation equivariance. Invariance comes from global pooling and from augmentation — and nothing gives rotation or scale invariance for free.
3. The receptive field is the design budget. Stride is what makes it grow geometrically; dilation buys reach without downsampling.
4. Two 3×3 layers beat one 5×5 — fewer weights, more nonlinearity. This is why 3×3 is the default.
5. The backward pass is three convolutions; used forwards, the input-gradient one is transposed convolution, and it checkerboards. Prefer upsample + convolve.
6. Of the classic architecture, what survived is 3×3 stacks, residuals, normalization, global average pooling, depthwise separable and 1×1 convolutions. The big FC heads, LRN and large stem kernels did not.
7. ConvNeXt shows much of the ViT advantage was the training recipe. The families converged.
8. Convolutions lost the ImageNet backbone race and kept the latent autoencoders, medical segmentation, on-device vision and the patch stem — which is most of the deployed pixels in the world.