13. Vision Transformers

The previous chapter built an encoder for sequences of text tokens. In 2021 Dosovitskiy et al. asked a deliberately blunt question: what if you feed an image to that same encoder, unchanged?¹ The answer — "An Image is Worth 16×16 Words" — was that with enough data a near-vanilla Transformer beats a CNN at image classification, with no convolutions anywhere.

The result mattered less as a classification score than as a demonstration: the architecture does not need to know it is looking at an image. That is why the ViT is the bridge chapter. Once you have it, the image encoders inside CLIP, inside every vision-language model, inside Stable Diffusion and DiT all stop being separate things to learn.

The inductive-bias trade, priced honestly

Chapter 10 called a convolution a strong, cheap, fixed prior. A self-attention layer has none of it: every patch may attend to every other patch from layer one, and any notion that neighbouring pixels are related has to be learned from data.

That is a trade, not an improvement, and the original paper measured both sides of it:

Read the table as a single sentence: the convolutional prior is a floor at small scale and a ceiling at large scale. Below some data threshold it gives you for free what the Transformer has to buy; above it, it forbids relationships the Transformer is free to discover.

The pipeline

The only genuinely new mechanism is the first step. Everything after it is the encoder you already know.

An image \(x \in \mathbb{R}^{H \times W \times C}\) is cut into non-overlapping \(P \times P\) patches, each flattened and projected by one shared linear layer:

For \(224 \times 224\) and \(P = 16\), that is \(N = 196\) tokens. Positional embeddings are added, because attention alone cannot tell the top-left patch from the bottom-right one, and the sequence goes through \(L\) standard pre-norm encoder blocks.

Patch size is the whole design decision

\(P\) controls two things at once, in opposite directions, and there is nothing else in a ViT with that much leverage.

Everything inside a patch is compressed into one \(d\)-dimensional vector before attention ever runs. No later layer can recover what the projection discarded. Drag the patch size to 32 and watch a face disappear into four tokens.

Pull it the other way and the count explodes: halving \(P\) quadruples \(N\), and the attention term grows with \(N^2\). Doubling the resolution does the same. This single quadratic is why:

• 14 and 16 are the near-universal patch sizes, and have been for five years;
• high-resolution vision-language models tile a large image into several 224- or 336-pixel crops rather than feeding one huge grid to the encoder;
• NaViT-style patch-and-pack⁹ exists at all — pack variable-resolution images into one sequence, keep the native aspect ratio, and skip the tokens you do not need.

What changed since 2021

The 2021 ViT is still recognizable, but a model you would download today differs in five places, and all five are worth knowing because they show up in the vision encoder of every multimodal model.

What a ViT is actually for in 2026

Very little of the deployed use is "classify this image into 1000 classes". The ViT won by becoming the standard way to turn pixels into tokens for something else:

CNN vs. ViT, updated

Implementation reference

import torch, torch.nn as nn

class PatchEmbed(nn.Module):
    """Image -> sequence of patch tokens. One Conv2d does patchify AND projection."""
    def __init__(self, patch=16, in_ch=3, dim=768):
        super().__init__()
        self.proj = nn.Conv2d(in_ch, dim, kernel_size=patch, stride=patch)

    def forward(self, x):                    # (B, C, H, W)
        x = self.proj(x)                     # (B, dim, H/p, W/p)
        return x.flatten(2).transpose(1, 2)  # (B, N, dim)

class ViT(nn.Module):
    """Registers instead of a CLS token, mean pooling instead of a CLS head."""
    def __init__(self, dim=768, depth=12, heads=12, n_classes=1000,
                 n_patches=196, n_registers=4):
        super().__init__()
        self.patch_embed = PatchEmbed(dim=dim)
        self.reg = nn.Parameter(torch.zeros(1, n_registers, dim))
        self.pos = nn.Parameter(torch.zeros(1, n_patches, dim))
        layer = nn.TransformerEncoderLayer(dim, heads, dim * 4, activation='gelu',
                                           norm_first=True, batch_first=True)
        self.encoder, self.norm = nn.TransformerEncoder(layer, depth), nn.LayerNorm(dim)
        self.head = nn.Linear(dim, n_classes)
        self.n_reg = n_registers

    def forward(self, x):
        x = self.patch_embed(x) + self.pos
        x = torch.cat([self.reg.expand(x.size(0), -1, -1), x], dim=1)
        x = self.encoder(x)[:, self.n_reg:]      # drop the registers; they were scratch space
        return self.head(self.norm(x).mean(dim=1))

import timm
# a supervised ViT, fine-tuned on your classes
model = timm.create_model('vit_base_patch16_224', pretrained=True, num_classes=10)

# DINOv2 features, frozen — usually the stronger option when labels are scarce
dino = torch.hub.load('facebookresearch/dinov2', 'dinov2_vitb14')
feats = dino.forward_features(img)['x_norm_patchtokens']   # (B, N, 768)

Key takeaways

1. A ViT is the text encoder applied to image patches. The only new part is patchify — and that is one strided convolution.
2. The convolutional prior is a floor at small scale and a ceiling at large scale. That is the whole trade.
3. The 2021 data threshold was largely a recipe problem. DeiT and ConvNeXt closed the gap from both directions.
4. Patch size is the design decision: information inside a patch is destroyed before attention runs, and \(N\) grows quadratically as \(P\) shrinks or resolution grows. 14–16 is where the compromise lands.
5. A current encoder uses pooling instead of [CLS], 2-D RoPE instead of learned positions, and register tokens to absorb the attention artifacts.
6. ViTs won because they emit tokens, which is what a language model eats. The classification score was never the point.
7. In production a ViT is usually the vision encoder of something else: a VLM, CLIP/SigLIP, DINOv2 features, or a diffusion backbone.