13. Vision Transformers

The previous chapter built an encoder for sequences of text tokens. In 2021 Dosovitskiy et al. asked a deliberately blunt question: what if you feed an image to that same encoder, unchanged?¹ The answer — "An Image is Worth 16×16 Words" — was that with enough data a near-vanilla Transformer beats a CNN at image classification, with no convolutions anywhere.

The result mattered less as a classification score than as a demonstration: the architecture does not need to know it is looking at an image. That is why the ViT is the bridge chapter. Once you have it, the image encoders inside CLIP, inside every vision-language model, inside Stable Diffusion and DiT all stop being separate things to learn.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. The three simulators and the three diagrams of this page run inside the slides. Open the slides in a new tab

The inductive-bias trade, priced honestly

Chapter 10 called a convolution a strong, cheap, fixed prior. A self-attention layer has none of it: every patch may attend to every other patch from layer one, and any notion that neighbouring pixels are related has to be learned from data. Follow one patch through both, layer by layer:

That is a trade, not an improvement, and the original paper measured both sides of it:

The same result model by model: five ViTs from the paper's appendix and the largest ResNet of BiT¹³, each pretrained on the three sets and fine-tuned on ImageNet.

Read the table as a single sentence: the convolutional prior is a floor at small scale and a ceiling at large scale. Below some data threshold it gives you for free what the Transformer has to buy; above it, it forbids relationships the Transformer is free to discover.

The pipeline

The only genuinely new mechanism is the first step. Everything after it is the encoder you already know.

An image \(x \in \mathbb{R}^{H \times W \times C}\) (\(H \times W\) pixels, \(C\) channels: 3 for RGB) is cut into non-overlapping \(P \times P\) patches. Each is flattened into \(P^2 C\) numbers and projected by one shared linear layer \(E\) to \(d\) numbers, the width of the model:

For \(224 \times 224\) and \(P = 16\), that is \(N = 196\) tokens. The 2021 ViT puts one more token in front, [CLS]: a learned vector, the same for every image, whose row at the output is the one the classification head reads. Positional embeddings are added, because attention alone cannot tell the top-left patch from the bottom-right one, and the sequence goes through \(L\) standard pre-norm encoder blocks⁴. Without positions the encoder is permutation-equivariant: shuffle the input tokens and the output tokens come out shuffled the same way, so neither a mean over them nor the [CLS] row can see the order (the jigsaw of the panel below).

Follow the tensor: ViT-B/16 at 224 px

The name reads model size / patch size: ViT-B/16 is the Base model (\(d = 768\), \(L = 12\), 12 heads) with \(P = 16\). One image, shape by shape:

Two things to notice. The sequence length never changes inside the encoder: every block maps \(197 \times 768\) to \(197 \times 768\). And almost all the parameters are in the blocks, which do not know the image size — only the position table does, which is why changing the resolution means interpolating it. Step through what that does to ViT-B/16 fine-tuned at 384 px, and how 2-D RoPE avoids it:

The whole pipeline, as code, on numbers small enough to check by hand: a 4 × 4 one-channel image, \(P = 2\) (four patches of four numbers), \(d = 3\) and one attention step of [CLS]. The vertical bar scores 0.436 for vertical; the same patches as a jigsaw score 0.393, and with the positions off both score 0.414. The checkerboard shows what the \(4 \to 3\) projection throws away.

Patch size is the whole design decision

\(P\) controls two things at once, in opposite directions, and there is nothing else in a ViT with that much leverage.

Everything inside a patch becomes one token before attention ever runs. Attention moves information between patches, never inside one: from the first layer on, the model works on a \(\frac{H}{P} \times \frac{W}{P}\) grid, and anything that only exists below that grid has to survive the projection \(E\). Whether it can depends on a ratio, \(P^2C\) numbers in and \(d\) out:

The panel draws each patch with its mean colour, the crudest summary a token could hold. Drag the patch size to 32 and watch the face shrink to a handful of tokens; the lab rebuilds an image from the patch embedding of a real pretrained ViT-Ti/16 to show what a \(768 \to 192\) projection actually keeps.

Pull it the other way and the count explodes: halving \(P\) quadruples \(N\). Doubling the resolution does the same. Per image, a ViT costs about

so quadrupling \(N\) quadruples the first term and multiplies the second by 16. At 224 px the first term dominates: for ViT-B/16 attention is 4% of the 17.4 G multiply-adds, and going to \(P = 8\) costs 4.5×, not 16×. The quadratic takes over only when \(N\) passes \(6d\) — 4,608 tokens for ViT-B — which is exactly the regime of high resolution: at 448 px with \(P = 8\), attention is already 40% of the bill, and the \(N \times N\) score matrix has to fit in memory too. That is why:

• 14 and 16 are the near-universal patch sizes, and have been since 2020;
• high-resolution vision-language models tile a large image into several 224- or 336-pixel crops rather than feeding one huge grid to the encoder;
• NaViT-style patch-and-pack⁹ exists at all — pack variable-resolution images into one sequence, keep the native aspect ratio, and skip the tokens you do not need.

What changed since 2021

The 2021 ViT is still recognizable, but a model you would download today differs in four places, and all four are worth knowing because they show up in the vision encoder of every multimodal model.

What a ViT is actually for in 2026

Very little of the deployed use is "classify this image into 1000 classes". The ViT won by becoming the standard way to turn pixels into tokens for something else:

CNN vs. ViT, updated

Implementation reference

import torch, torch.nn as nn

class PatchEmbed(nn.Module):
    """Image -> sequence of patch tokens. One Conv2d does patchify AND projection."""
    def __init__(self, patch=16, in_ch=3, dim=768):
        super().__init__()
        self.proj = nn.Conv2d(in_ch, dim, kernel_size=patch, stride=patch)

    def forward(self, x):                    # (B, C, H, W)
        x = self.proj(x)                     # (B, dim, H/p, W/p)
        return x.flatten(2).transpose(1, 2)  # (B, N, dim)

class ViT(nn.Module):
    """Registers instead of a CLS token, mean pooling instead of a CLS head."""
    def __init__(self, dim=768, depth=12, heads=12, n_classes=1000,
                 n_patches=196, n_registers=4):
        super().__init__()
        self.patch_embed = PatchEmbed(dim=dim)
        # random init: identical registers would get identical gradients and stay identical
        self.reg = nn.Parameter(torch.randn(1, n_registers, dim) * .02)
        self.pos = nn.Parameter(torch.randn(1, n_patches, dim) * .02)
        layer = nn.TransformerEncoderLayer(dim, heads, dim * 4, dropout=0.0, activation='gelu',
                                           norm_first=True, batch_first=True)
        self.encoder = nn.TransformerEncoder(layer, depth, enable_nested_tensor=False)
        self.norm = nn.LayerNorm(dim)
        self.head = nn.Linear(dim, n_classes)
        self.n_reg = n_registers

    def forward(self, x):
        x = self.patch_embed(x) + self.pos
        x = torch.cat([self.reg.expand(x.size(0), -1, -1), x], dim=1)
        x = self.encoder(x)[:, self.n_reg:]      # drop the registers; they were scratch space
        return self.head(self.norm(x).mean(dim=1))

import timm
# a supervised ViT, fine-tuned on your classes
model = timm.create_model('vit_base_patch16_224', pretrained=True, num_classes=10)

# DINOv2 features, frozen — usually the stronger option when labels are scarce
dino = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitb14')
feats = dino.forward_features(img)['x_norm_patchtokens']   # (B, N, 768)

Lab: the weak prior, measured

Every claim on this page is measured in a Colab notebook on CIFAR-10 (60,000 colour images, 32 × 32, ten classes). It runs in the browser on a free T4 GPU — under fifteen minutes of compute; there is nothing to install. Nine sections, each ending in a checkpoint (the first eight with a check value):

1. patchify written by hand and checked against F.unfold;
2. the patch embedding as a strided convolution, and what a pretrained ViT-Ti/16's \(768 \to 192\) projection keeps of each patch;
3. a jigsaw-shuffled image that a ViT without positions cannot tell from the original;
4. the price of a token, from the formula and from the clock;
5. a small CNN against a TinyViT on 1,000, 5,000 and 20,000 images, with the same budget;
6. the same at 5,000 images without augmentation — the recipe, at small scale;
7. a frozen pretrained ViT-Ti/16 with a logistic regression on top, against the same architecture with random weights;
8. what the pretrained ViT learned instead of the prior: the 2-D grid in its position table and the attention distance of each head;
9. a ViT of your own for 5,000 images, under a 600,000-parameter budget.

Open the lab in Colab — save a copy to your Drive (File → Save a copy in Drive) to run and edit it.

Key takeaways

1. A ViT is the text encoder applied to image patches. The only new part is patchify — and that is one strided convolution.
2. The convolutional prior is a floor at small scale and a ceiling at large scale. That is the whole trade.
3. The 2021 data threshold was largely a recipe problem. DeiT and ConvNeXt closed the gap from both directions.
4. Patch size is the design decision: a patch is one token before attention runs, so nothing below the patch grid is seen except through the projection, and \(N\) grows quadratically as \(P\) shrinks or resolution grows. 14–16 is where the compromise lands; the attention term dominates the cost only once \(N > 6d\).
5. A current encoder uses pooling instead of [CLS], 2-D RoPE instead of learned positions, and register tokens to absorb the attention artifacts.
6. ViTs won because they emit tokens, which is what a language model eats. The classification score was never the point.
7. In production a ViT is usually the vision encoder of something else: a VLM, CLIP/SigLIP, DINOv2 features, or a diffusion backbone.
8. At small scale the weak prior is a real price. In the lab, on 1,000 to 20,000 CIFAR-10 images, a small CNN beats a ViT of similar compute by 15 to 20 points; a frozen ViT pretrained on 14M images beats both with 1,000 labels and a linear probe (a logistic regression on its frozen features). The prior can be learned — from far more data.