Artificial Neural Networks and Deep Learning · Insper

Vision Transformers

The weak prior, priced

A deliberately blunt question

The previous class built an encoder for sequences of text tokens.
What happens if you feed it an image, unchanged?

  • First problem: an image is not a sequence. Cut it into patches and read them like words.
  • Dosovitskiy et al., An Image is Worth 16×16 Words (2021): with enough data, a near-vanilla Transformer beats a CNN at classification.
  • The score mattered less than the demonstration: the architecture does not need to know it is looking at an image.

Why this is the bridge class

Once you have the ViT, these stop being separate things to learn:

  • the image tower of CLIP and SigLIP
  • the vision encoder of every vision-language model
  • the backbone of DiT — a ViT on latent patches
  • DINOv2 features, used frozen across robotics and medicine

A ViT is the standard way to turn pixels into tokens for something else.

The inductive-bias trade

Convolution — a strong, cheap, fixed prior: local, the same everywhere.

Self-attention — none of it: every patch may attend to every patch from layer 1. "Neighbours are related" has to be learned from data.

pretraining data ViT vs. ResNet
ImageNet-1k (1.3M) ViT loses
ImageNet-21k (14M) roughly level
JFT-300M (300M) ViT wins

The convolutional prior is a floor at small scale and a ceiling at large scale.

One patch, two models

One patch of a 14 by 14 grid, in two models. Left, 3 by 3 convolutions on the grid: the receptive field grows by one patch on each side per layer and reaches every patch only at layer 7. Right, self-attention: every patch is reachable from layer 1. The mean attention distances measured in a pretrained ViT-Ti/16 are 73, 84 and 13 pixels in layer 1 and 145, 142 and 145 pixels in layer 12. CNN · 3 × 3 CONVOLUTIONS ON THE SAME GRID VIT · SELF-ATTENTION 1 2 3

The table, model by model

ImageNet top-1 accuracy after fine-tuning against the size of the pretraining set: ImageNet-1k, ImageNet-21k and JFT-300M. ResNet BiT R152x4: 81.3, 85.4, 87.5. ViT-B/32: 73.4, 81.3, 80.7. ViT-B/16: 77.9, 84.0, 84.2. ViT-L/32: 71.2, 81.0, 84.4. ViT-L/16: 76.5, 85.2, 87.1. ViT-H/14: 85.1 and 88.0 with the two larger sets.

The threshold moved

"ViT needs 300M images" was mostly "ViT needs a training recipe nobody had written yet".

  • DeiT (2021): augmentation, regularization and a distillation token — a competitive ViT on ImageNet-1k alone.
  • Steiner et al.: the right augmentation and regularization, with more compute, are worth about as much as 10× more data.
  • ConvNeXt (2022): the same point from the other side — a ResNet modernized with the recipe and design choices of the ViTs matches Swin, a hierarchical ViT.

The trade is real. The 2021 crossover point was not.

distillation token: one more token, trained to match the prediction of a CNN teacher

Simulator · the pipeline

Patchify, in symbols

H × W pixels, C channels (3 for RGB) · P: patch side · N: number of patches = number of tokens · d: numbers per token · E: the linear projection, P²C → d

A 224 × 224 RGB image with : how many tokens, and how many numbers in each flattened patch?

  • tokens
  • each patch: numbers — equal to ViT-B's by coincidence: that came from BERT-Base

Follow the tensor: ViT-B/16

ViT-B/16 = the Base model (, blocks, 12 heads) with patch . One image:

step shape parameters
input (channels × height × width) 3 × 224 × 224 —
patches, flattened (tokens × numbers) 196 × 768 —
projection = Conv2d(3, 768, 16, stride=16) 196 × 768
prepend [CLS], add positions 197 × 768
12 encoder blocks 197 × 768 M
final LayerNorm, read the [CLS] row 768
linear head 1000
total ≈ 86.6M

[CLS]: a learned token in front of the patches; the head reads its row · Conv2d(channels in, filters, kernel side, stride) · one block = 7,087,872 (GPT-2 small's): 2,362,368 attention + 4,722,432 MLP (768 → 3072 → 768) + 3,072 LayerNorms

The sequence length never changes inside the encoder, and only the position table knows the image size.

The patch embedding is a convolution

  • A kernel with stride reads each patch exactly once.
  • Its output channels are the rows of .
  • Every implementation does it this way: one op that patchifies and projects.

, (the lab's TinyViT, CIFAR 32 × 32): how many parameters?

  • Lab: Conv2d and Linear on flattened patches agree up to rounding (1e-6 at most).

"No convolutions" has an asterisk: there is exactly one, at the entrance. Only it and the interpolation of slide 14 build the 2-D layout in by hand.

Why positions are needed

Shuffle the patches of a CIFAR image like a jigsaw. Does a ViT without positional embeddings notice?

  • Attention mixes tokens by content, not place; the MLP and LayerNorm act on each token alone.
  • So the encoder is permutation-equivariant, and mean pooling forgets the order entirely.
  • Lab, untrained TinyViT, largest change in the logits: 1e-7 (rounding) without positions, 9e-3 with them — small because the table starts with a standard deviation of 0.02, but real.
  • Without positions, the original and the jigsaw are the same image. A CNN never has this problem.

Simulator · a ViT, line by line

x · image 4 × 4, 1 channel (edit it)

A new resolution: the position table

The learned position table of ViT-B/16 at a new resolution. Trained at 224 pixels it has one row for the CLS token and 14 by 14 rows for the patches. At 384 pixels the image has 24 by 24 patches. The 196 patch rows are put back on their 14 by 14 grid and interpolated to 24 by 24; each new position is a weighted mean of four old ones. 2-D RoPE has no table: it rotates queries and keys by the row and column of each patch. [CLS] [CLS] ?

Patch size is the whole design decision

A patch becomes one token before attention runs. Attention moves information between patches, never inside one.

model what can keep
ViT-B/16 everything, in principle ( is square)
ViT-Ti/16 (Tiny, ) at most of the directions
ViT-B/32 at most

Lab: rebuilding each patch from a pretrained ViT-Ti/16 token (the best linear way back, the pseudoinverse of ) leaves a relative error of 0.37 — lost before the first attention layer.

Simulator · patch size and cost

The price of a token

L blocks, N tokens, d numbers per token · per block, 12Nd² = 4Nd² for the Q, K, V and output projections + 8Nd² for the MLP d → 4d → d · 2N²d = QK⊤ + the weighted sum of V, N²d each

ViT-B/16 at 224 px (, ). What fraction of the cost is attention?
Halve the patch to 8: does the cost grow 4× or 16×?

The price of a token · answer

ViT-B/16 (, ), in G (billions of) multiply-adds per image:

image / patch total attention
224 / 16 16.65 0.71 17.36 4%
224 / 8 66.59 11.33 77.92 15%
448 / 8 266.36 181.27 447.62 40%
  • Halving : × 4, the first term × 4 and the second × 16. Since the first dominates, the total goes × 4.5 (), not × 16.
  • Attention overtakes the rest only when , that is, = 4,608 tokens for ViT-B.

So, in practice

  • 14 and 16 have been the near-universal patch sizes since 2020.
  • Doubling the resolution also quadruples (): high-resolution vision-language models (VLMs) tile a large image into 224- or 336-px crops instead of one huge grid.
  • NaViT patch-and-pack: variable-resolution images in one sequence, native aspect ratio, skip the tokens you do not need.

Small keeps detail and pays in tokens; large is cheap and throws detail away before attention. There is nothing else in a ViT with that much leverage.

What changed since 2021 (1/2)

[CLS] → pooling

  • Reading the class from one special token was a BERT habit.
  • Many encoders average the patch tokens or use an attention-pooling head (SigLIP's: one learned query attends over the patches).
  • In a VLM there is no head at all: the patch tokens go to the LLM.

learned positions → 2-D RoPE

  • A learned table must be interpolated when the resolution changes, at a cost in accuracy.
  • 2-D RoPE: the rotation of the attention class, on rows and columns. Relative, no table to interpolate.
  • Qwen2-VL, DINOv3.

What changed since 2021 (2/2)

register tokens

  • Large, well-trained ViTs dump attention on a few meaningless background patches — scratch space.
  • It wrecks the attention maps and hurts dense tasks (an answer per pixel: segmentation, depth).
  • Add a few unused tokens; discard them at the output. The artifacts vanish (Darcet et al., 2024).

hierarchy, where needed

  • A plain ViT keeps one resolution throughout — wasteful for detection and segmentation.
  • Swin: windowed attention and a pyramid.
  • ViTDet: a plain ViT with a simple feature pyramid on top.

What a ViT is for in 2026

use what the ViT provides
vision-language models the image encoder: patch tokens, projected, prepended to the text
CLIP / SigLIP the image tower, trained against a text tower
self-supervised features DINOv2/v3: dense features without labels, used frozen
generative backbones DiT, MMDiT: a ViT on latent patches
masked pretraining MAE: mask 75%, encode only the visible 25%

Very little of it is "classify into 1000 classes".

The reason it won is not accuracy

A ViT and a modern ConvNet score about the same on ImageNet. Why is the ViT everywhere?

  • It emits a sequence of tokens, in the format text uses.
  • Concatenating patch tokens with word tokens and running one Transformer over both is trivial.
  • Doing the same with a convolutional feature map is not.

The argument is multimodality, not classification.

CNN vs. ViT, updated

CNN ViT
core operation convolution: local, fixed self-attention: global, learned
inductive bias strong weak — bought with data
receptive field grows with depth global at layer 1
cost +
small data wins needs pretraining or a recipe
web-scale data keeps up with the same recipe where pretraining happens
multimodal use awkward native

Measured: the floor at small scale

CIFAR-10, same budget (600 steps with batches of 128 images, AdamW, crop + flip). Test accuracy:

training images small CNN (289k parameters) TinyViT (546k) CNN − ViT
1,000 0.58 0.43 0.15
5,000 0.68 0.51 0.17
20,000 0.71 0.51 0.20
pretrained ViT-Ti/16, linear probe, 1,000 labels 0.89
  • The CNN wins at every size, by 15, 17 and 20 points, with half the parameters — and at this budget the gap does not close.
  • The prior can be bought elsewhere: pretrained on 14M images (ImageNet-21k), a frozen ViT plus a logistic regression on its 192 features (the linear probe) beats both. Random weights, same architecture: 0.33.

Measured: part of the gap is the recipe

Same 5,000 images, without crop and flip:

with augmentation without loss
small CNN 0.68 0.68 0.001
TinyViT 0.51 0.47 0.04

The accuracies are rounded to two decimals; the loss is the difference before rounding.

Augmentation teaches the ViT part of what the CNN is handed. That is DeiT's lesson, at small scale.

What the ViT learned instead of the prior

Inside the pretrained ViT-Ti/16 (, 12 layers of 3 heads, a grid of 14 × 14 patches):

Positions. 196 = 14 × 14 free vectors, nothing says which is next to which.

  • cosine similarity to the 4 neighbours: 0.85
  • at grid distance ≥ 8 patches: −0.17
  • the 2-D grid, recovered from data

Attention distance: how far a patch looks, on average weighted by attention (1 patch = 16 px):

  • layer 1, three heads: 73, 84, 13
  • layer 12: 145, 142, 145
  • one early head is a learned local filter; the others already look far. Depth makes all global.

Lab · the weak prior, measured

Colab notebook, ~2 h, CIFAR-10, T4 GPU (under 15 min of compute). Each section ends in a checkpoint.

  1. patchify, by hand
  2. the patch embedding is a strided conv — and what 768 → 192 keeps
  3. without positions: a bag of patches
  4. the price of a token
  5. the data hunger: CNN × ViT
  1. part of the gap is the recipe
  2. a prior bought elsewhere: linear probe
  3. what the pretrained ViT learned
  4. your ViT for small data: ≤ 600,000 parameters, 5,000 images

open in Colab →

Takeaways

  1. A ViT is the text encoder applied to image patches. The only new part is patchify — one strided convolution.
  2. Without positions, attention sees a bag of patches.
  3. Patch size is the decision: nothing below the patch grid is seen except through ; halving quadruples . Attention dominates only once .
  4. The convolutional prior is a floor at small scale and a ceiling at large scale; recipe and pretraining move the crossover.
  5. Current encoders: pooling instead of [CLS], 2-D RoPE, registers.
  6. ViTs won because they emit tokens — what a language model eats.

Outline: (1) the question and the inductive-bias trade; (2) the pipeline: patches, the patch embedding, positions; (3) patch size: what a token keeps and what it costs; (4) what changed since 2021 and what a ViT is for; (5) what the lab measured. Closes with the Colab lab.

Six steps on a 14 × 14 grid. Layer 1: the 3×3 kernel sees 9 of 196, attention all 196. Layer 3: 7 × 7 = 49. Layer 7: the whole grid (a real CNN downsamples and gets there sooner, but its first layers stay local). Before step 5, ask: in layer 1 of a trained ViT, do the heads look near or far? Lab, ViT-Ti/16: 73, 84 and 13 px; layer 12: 145, 142 and 145 px, farther than uniform attention (116 px).

Table 5 of the ViT paper and BiT's ResNet-152x4, all fine-tuned on ImageNet. Point at ImageNet-1k first: every ViT below the ResNet, and L/16 (76.5) below B/16 (77.9). Hover L/16 and follow it up: level at 21k, near the top at JFT. H/14 passes the ResNet at JFT (88.04 against 87.54) with about a quarter of the pretraining compute (2.5k against 9.9k TPUv3-core-days).

Step through it. The only new step is the first one (patchify + linear projection); from "Transformer encoder" on it is the previous class. The [CLS] + MLP head is the 2021 ViT; current encoders mostly average the patch tokens.

A 4×4 image, P = 2, d = 3: every number checks by hand. Step through the vertical bar: patchify gives four rows of four numbers, E keeps the mean and two edges, [CLS] attends with weights 0.273, 0.243, 0.162, 0.193, 0.129 and the head says vertical (0.436). On the last line point at the jigsaw: 0.393, different, because of the positions. Then "no positions": both 0.414, the bag of patches of the previous slide. Last, the checkerboard: lost = 1 in every patch, and the tokens equal a uniform grey.

Six steps. The table of ViT-B/16 is 197 × 768. At 384 px there are 576 patches and only 196 rows. Put the rows back on the 14 × 14 grid, stretch it to 24 × 24 (each new row a weighted mean of four old ones: 0.27, 0.44, 0.11, 0.18 for row 6, column 10), keep [CLS]: 577 × 768, then fine-tune. Last step: 2-D RoPE, no table at all. Hover the cells.

Drag P to 32 at 224 px: the face becomes ~9 tokens and 75% of each patch's dimensions are gone with ViT-B. Then P = 8: 784 tokens. Ask for the cost ratio before reading it (next slide).