Artificial Neural Networks and Deep Learning · Insper

Convolutional Neural Networks

The constraint, priced

Start with the arithmetic

A 224 × 224 × 3 photo goes into a dense layer of 1,000 neurons.
How many weights does that layer have?

  • inputs 1,000 outputs 150 million weights — in the first layer alone.
  • And the number is not the worst part: the edge detector in the corner and the one in the centre are different weights, learned separately, from data you do not have.

Two claims about images

  1. What matters is local. A pixel talks to its neighbours, not to the opposite corner.
  2. It means the same thing anywhere. An edge in the corner is the same event as an edge in the centre.

A convolutional layer is a dense layer with those two things imposed on it:
locality and weight sharing. Nothing is added — capacity is removed, on purpose.

An architecture is a prior written as a constraint on the weights.

The same layer, three times

6 inputs → 4 outputs (1-D, kernel 3). Each cell is a weight; the colour says which ones are the same number.

dense
24 distinct weights
0 0 0 0 0 0 0 0 0 0 0 0
+ locality
12 distinct weights
w1 w2 w3 0 0 0 0 w1 w2 w3 0 0 0 0 w1 w2 w3 0 0 0 0 w1 w2 w3
+ sharing
3 weights — a convolution

One output position = one dot product

stride 1, no padding

  • Corner window:
  • All four windows:
  • The same four numbers of were read at every position.

PyTorch computes a cross-correlation (it does not rotate the kernel). For the network it does not matter: the kernel is learned. For checking derivatives by hand it does — it comes back in the backward pass.

Simulator · the sliding kernel

The output shape

  • Stride : how far the window jumps. halves the resolution — the cheapest downsampling.
  • Padding : invented zeros at the border. With odd , is "same" padding: the output keeps its size.
  • Pooling follows the same formula (usually , ).

Almost every CNN bug is a shape bug. When PyTorch complains at the first Linear, print the shape layer by layer — do not guess the number.

Exercise · follow the tensor

Input (1, 1, 28, 28) = (images, channels, height, width): one grayscale 28×28 image. Fill it in before the next slide.

layer output
Conv2d(1, 8, 5) ?
MaxPool2d(2) ?
Conv2d(8, 16, 3, padding=1) ?
MaxPool2d(2) ?
Conv2d(16, 32, 3, stride=2) ?
Flatten() ?

Conv2d(channels in, number of filters, kernel side)

MaxPool2d(2): 2×2 window, step 2

Output side:

Answer as channels × height × width.

Follow the tensor · answer

1×28×28conv 5×58 filters8×24×24max pool 28×12×12conv 3×3, p=116 filters16×12×12max pool 216×6×6conv 3×3, s=232 filters32×2×2flatten128

layer output where it comes from
Conv2d(1, 8, 5) 8 × 24 × 24 8 filters · (28 + 0 − 5)/1 + 1 = 24
MaxPool2d(2) 8 × 12 × 12 same channels · (24 − 2)/2 + 1 = 12
Conv2d(8, 16, 3, padding=1) 16 × 12 × 12 16 filters · (12 + 2 − 3)/1 + 1 = 12
MaxPool2d(2) 16 × 6 × 6 same channels · (12 − 2)/2 + 1 = 6
Conv2d(16, 32, 3, stride=2) 32 × 2 × 2 32 filters · ⌊(6 + 0 − 3)/2⌋ + 1 = 2
Flatten() 128 32 · 2 · 2 → the next one is Linear(128, …)

From one kernel to a layer: channels

  • The input has channels (3 for RGB, 64 after a layer…).
  • Each kernel is a block: it reads every channel and adds it all into one number.
  • kernels → feature maps, which become the channels of the next layer.

Conv2d(3, 64, 3) on a 32 × 32 image. How many parameters?

  • The equivalent dense layer (): ≈ 201 million
  • There is no and no in the count: the same layer runs on 32 × 32 or on 4K.

Equivariance is what you get…

Sharing weights gives an exact property: shift the input and the map shifts with it.

That is translation equivariance — exactly right for a detector.

…invariance is what you wanted

A classifier wants "ankle boot" wherever the boot is. Equivariance only becomes invariance when:

mechanism what it does
pooling / strided conv discards position within the window — a small, local invariance
global average pooling at the end collapses the whole — this is where invariance comes from
data augmentation buys, with examples, invariance to what the conv does not cover

What the convolution does not give

The prior covers translation and nothing else. Rotation, scale, viewpoint: bought with augmentation and data — in a CNN exactly as in a Transformer.

  • Not even translation is exact: every stride and every pooling samples the grid. A shift that is not a multiple of the total stride does not come back identical at the top.
  • Flatten collapses nothing: it keeps each position in its own coordinate of the vector.

In the lab: an MLP, a CNN + Flatten and a CNN + GAP trained on Fashion-MNIST. Shift the test set by 4 px. Which one drops least? And rotated by 90°?

Measured in the lab

Three training epochs on Fashion-MNIST, Adam, lr = 1e-3. Test accuracy:

network parameters in place shifted 4 px shifted 8 px rotated 90°
MLP 784 → 256 → 128 → 10 235,146 0.86 0.25 0.18 0.05
CNN + Flatten 20,586 0.90 0.33 0.19 0.06
CNN + GAP 24,170 0.85 0.81 0.65 0.07
CNN + Flatten, augmentation ±4 px 20,586 0.85 0.80 0.26 —
  • GAP buys invariance — and charges a few points in the usual place: throwing position away costs information.
  • Augmentation buys invariance only within the range it showed.
  • Nothing buys rotation for free.

The receptive field is the budget

A deep unit does not see the image: it sees whatever fed it, recursively.

  1. Stride 1: linear growth. Each 3×3 adds 2. To see 224 px: 112 layers.
  2. Stride 2: geometric growth. Each stride doubles the jump of everything above — that is why CNNs downsample.
  3. Dilation: reach without weights and without losing resolution — semantic segmentation.

Self-attention reaches every position at layer 1, at an O(n²) cost. The whole CNN × Transformer trade-off is in that sentence.

Simulator · receptive field

Two 3×3 beat one 5×5

Which one sees more? Which one costs less?

  • Both reach 5 pixels.
  • Two 3×3: weights. One 5×5: .
  • And the pair puts a nonlinearity in between.
  • That was the whole VGG paper.
  • It is why 3×3 is the near-universal default…
  • …and why 11×11 stem convolutions (AlexNet) disappeared after 2014.

The way back: the same operation three times

gradient what it is implementation
sum of over batch and space a reduction
cross-correlation of the input with a convolution
full cross-correlation of with the 180°-rotated kernel (= full convolution with the kernel) a convolution (the "transposed" one)

With stride s > 1: insert s − 1 zeros between the entries of ∂L/∂Y and apply the same recipe.

The way back, with numbers

arrives for the earlier example ( 3×3, 2×2).

  • Bias: sum it all →
  • Kernel: →
  • Input: full convolution →
  • 2×2 came in, 3×3 goes out: each pixel gets the sum over every window it took part in — the corner in one, the centre in four. That is why the padding is on each side.

Simulator · the convolution code, line by line

X
K
b
dY

Transposed convolution and the checkerboard

  • The operation, used forwards, is ConvTranspose2d: the upsampling of decoders and GANs.
  • When the stride does not divide the kernel, some output positions receive more contributions than others.
  • Result: a visible checkerboard in the generated image.

The fix everybody uses:
nn.Upsample + Conv2d — never a bare transposed convolution.

Odena, Dumoulin & Olah (2016), Deconvolution and Checkerboard Artifacts, Distill.

Pooling: who receives the gradient

No weights. It summarizes each window into one number — and the choice of summary decides who receives gradient.

  • Max is a router, not a filter: the whole gradient goes to the window's winner; the rest gets zero.
  • Average splits it evenly: each cell gets — steadier, less selective.
  • The route changes during training: the winner depends on the values, not the indices.

6 × 6 input, max 2 × 2, stride 2, gradient 1 at every output. How many of the 36 cells receive anything?

Simulator · pooling, forward and back

Flatten × global average pooling

Flatten unrolls into a vector and hands it to a dense layer.

  • VGG-16: numbers → Linear(25088, 4096)
  • 102.8 million parameters — 74% of the whole network (138 M) in one layer.
  • And each weight tied to one position.

GAP averages each channel over .

  • numbers, no parameters.
  • The head becomes Linear(C, classes).
  • It collapses space → this is where invariance comes from.

A whole CNN, layer by layer

layer output parameters receptive field
input 1 × 28 × 28 — 1
Conv2d(1, 16, 3, padding=1) + BatchNorm + ReLU 16 × 28 × 28 192 3
MaxPool2d(2) 16 × 14 × 14 0 4
Conv2d(16, 32, 3, padding=1) + BatchNorm + ReLU 32 × 14 × 14 4,704 8
MaxPool2d(2) 32 × 7 × 7 0 10
Conv2d(32, 64, 3, padding=1) + BatchNorm + ReLU 64 × 7 × 7 18,624 18
AdaptiveAvgPool2d(1) + Flatten 64 0 the whole image
Linear(64, 10) 10 650 —
total 24,170

Resolution goes down, channels go up · the parameters live at the end · an MLP of similar accuracy has 235,146.

How the architecture got its shape

✅ survived

  • stacked 3×3 kernels (VGG)
  • residual blocks (ResNet) — now in every architecture
  • normalization after each conv
  • global average pooling instead of the dense head
  • depthwise separable and 1×1 convolutions

❌ did not

  • Local Response Normalization (AlexNet)
  • dense heads with tens of millions of parameters
  • large stem kernels (11×11 stride 4, 7×7 stride 2)
  • auxiliary classifiers (GoogLeNet)
  • sigmoid and tanh in vision stacks
  • the ImageNet backbone race itself

The cheap convolutions: 1×1 and separable

1×1 — a dense layer per position: mixes channels, ignores neighbours. It is how every architecture changes channel count cheaply.

Depthwise separable = one kernel per channel (depthwise) + a 1×1 (pointwise).

64 → 128, 3×3 parameters
Conv2d(64, 128, 3) 73,856
depthwise Conv2d(64, 64, 3, groups=64) 640
pointwise Conv2d(64, 128, 1) 8,320
separable 8,960 (12%)

Prediction: 1/128 + 1/9 ≈ 0.119. Every model that runs on a phone (MobileNet, EfficientNet) uses it.

ConvNeXt: the control experiment

2020–2021: "attention beat convolution". Did it?

  • Take a ResNet-50 and change only what is not attention: AdamW, 300 epochs, heavy augmentation, stage ratios, a 7×7 depthwise kernel, LayerNorm, GELU.
  • No attention anywhere.
  • Result: it matches Swin Transformer at every scale.

Much of the reported ViT advantage was the training recipe, not the architecture. The two families converged on the same block, from opposite directions.

Where convolutions live in 2026

still convolutional — and not by inertia

  • the VAE of every latent diffusion model (SD, SDXL, SD3, FLUX)
  • medical segmentation (nnU-Net): hundreds of volumes, not billions of images
  • on-device and real-time vision
  • audio, spectrograms, time series (1-D conv)
  • a ViT's patch embedding is a strided conv

now usually a Transformer

  • classification and retrieval at scale
  • open-vocabulary recognition (CLIP)
  • the text-to-image generation backbone (DiT, MMDiT)

The rule that generalizes

Convolution
a strong, cheap, fixed prior

wins when data is scarce, latency matters or the structure is genuinely local

Transformer
a weak, expensive, learnable prior

wins when you can pay for enough data to learn a better structure than the one you would have imposed

That trade-off is the axis the next four classes move along: attention, Transformers, ViT and transfer learning.

Lab · the constraint, measured

Colab notebook, ~2 h, Fashion-MNIST. Each section ends in a checkpoint with a check value.

  1. the convolution written by hand
  2. shapes — and the Linear error
  3. the price in parameters
  4. equivariance × invariance, measured
  5. the receptive field through the gradient
  1. the way back: three convolutions and a router
  2. MLP × CNN, on the shifted test set
  3. what the first layer learned
  4. your own CNN on a budget: ≤ 30,000 parameters, ≤ 5 epochs

open in Colab →

Takeaways

  1. Convolution = dense with locality and sharing. Parameters: , no and no .
  2. Shared weights give equivariance. Invariance comes from GAP and augmentation — rotation and scale, never for free.
  3. The receptive field is the budget: stride makes it grow geometrically; dilation, without losing resolution.
  4. Two 3×3 > one 5×5. The way back is three convolutions; used forwards, the transposed one makes a checkerboard.
  5. Max pooling is a router: one winner per window gets everything.
  6. ConvNeXt: much of the ViT advantage was the recipe. The families converged.
  7. Convolutions lost ImageNet and kept latent VAEs, medicine, phones and the patch stem.

Outline: (1) why restrict a dense layer; (2) shape and parameters; (3) equivariance and receptive field; (4) the way back: convolution and pooling; (5) the architecture that survived and where convolutions live in 2026. Closes with the Colab lab.

Let the class guess before revealing. The second bullet matters more than the count: the dense layer does not know that an edge is an edge anywhere.

Row i = output i, column j = input j. Dense: everything connected to everything. Locality: only the diagonal band is left. Sharing: the band uses the same three numbers in every row — a Toeplitz matrix. Multiplying by it is exactly sliding the kernel.

Press play with "vertical edge". Then set stride to 2 and padding to same, and ask for the formula before looking at the line below. The dense/conv ratio is the argument of the class.

In the lab (section 2) the same exercise comes with the Linear(256, 10) error.

Both CNNs have BatchNorm: the only difference between them is the head. Notebook, section 7.

Ask for predictions: 5 layers 3x3 stride 1 → 11. Stride 2 → ? (63). Dilation 2 → 21.

Default case = the previous slide's numbers. Step line by line; on line 4 click Y[1,1] (7), on line 7 the centre of dXp (four windows, 1) and then a corner (one window). Then: "count the windows" (dX = 1 2 1 / 2 4 2 / 1 2 1), "same" padding (the corner output's gradient mostly lands on the zeros and is dropped on line 8) and stride 2 (windows do not overlap: dX is copies of K scaled by dY).

Answer to the previous question: 9 receive, 27 get nothing. Then drag the stride to 1 and show the accumulation when windows overlap.