19. CLIP

Every generative model in the previous chapters could produce an image. None of them could produce the image you asked for, because none of them had any idea what a sentence means. CLIP (Contrastive Language–Image Pre-training) is where that changes, and it is why this chapter sits in the middle of a module about generation despite not being a generative model at all.

The idea² is almost aggressively simple. Take 400 million (image, caption) pairs off the internet. Train an image encoder and a text encoder — two separate networks, with no shared weights — jointly, under one instruction: put an image and its own caption close together, and everything else far apart.

That is the whole objective. What comes out of it is a single vector space that images and sentences both live in, and therefore an interface: you can now ask questions across the modality boundary. "How close is this picture to the phrase a photo of a dog?" becomes a dot product.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. The simulators and diagrams of this page run inside the slides. Open the slides in a new tab

The components

• Image encoder — a ViT or a modified ResNet (the paper trains both; the best model is a ViT-L/14 at 336 px: L = Large, 24 layers of width 1024; /14 = patches of 14 × 14 pixels), producing one fixed-size vector per image.
• Text encoder — a Transformer over at most 77 tokens, producing one vector per caption.
• A linear projection and an L2 normalization on each side. The two encoders do not even have the same width; the projections map both into the same \(d\)-dimensional space, and the normalization puts every embedding on the unit sphere, so cosine similarity is just a dot product. That shared space is the entire architectural commitment.
• A contrastive loss (symmetric InfoNCE: a softmax cross-entropy that must pick the right partner among the \(N\) pairs of the batch) over a batch: the diagonal of the image×text similarity matrix is right, everything else is wrong.

For openai/clip-vit-base-patch32, the cheapest of the released ViT models (50 tokens per image) and the one the lab uses, one batch of \(N\) pairs flows like this:

[CLS] is an extra learned token whose output summarizes the image (chapter 13). The text tower pools the end-of-text token instead because it is causal (each token sees only the ones before it): the last token is the only one that has seen the whole caption.

About 87.5 million parameters on the image side and 63.2 million on the text side. The towers share no weight: they meet only in the loss, through the dot product and the temperature.

The loss, and the two design decisions inside it

With \(I\) and \(T\) the \(N \times d\) matrices of normalized image and text embeddings (row \(i\) of each comes from the same pair), \(\tau > 0\) the temperature (below) and \(\text{CE}(Z, \text{diag})\) the softmax cross-entropy of each row of \(Z\) whose right class is the diagonal one (row \(i\) → column \(i\)):

Row \(i\) of \(I T^\top / \tau\) is a classification problem with \(N\) classes: which of the \(N\) captions belongs to image \(i\)? The correct answer is class \(i\). The second term asks the same question from each caption's side, over the columns.

Temperature \(\tau\) scales the similarities before the softmax. Small \(\tau\) makes the distribution sharp, so the loss concentrates on the hardest negative (the wrong caption with the highest similarity); large \(\tau\) treats all negatives alike and the representation stops sharpening. CLIP learns \(\tau\) rather than fixing it, so the model tunes its own difficulty as training goes. It is stored as a log-scale, starts at \(\tau = 0.07\) and is clipped so that \(1/\tau \le 100\); the released models end exactly at that clip, \(\tau = 0.01\). Why so low: the cosines of a right and a wrong caption differ by a few hundredths (0.26 against 0.20 in the lab), and only multiplying by \(1/\tau \approx 100\) turns that gap into a confident softmax. Drag it in the panel: notice that retrieval accuracy does not change — \(\tau\) affects the loss, not the ranking.

Batch size is part of the objective. Every other caption in the batch is a negative, so a batch of 32,768 gives each example 32,767 negatives. A model that guesses blindly pays \(\log N\) (natural log) per example, so a bigger batch is a harder task, not just a faster one: the more competitors, the finer the distinctions the encoders must learn. It also meant the similarity matrix had to be sharded across hundreds of accelerators, and that engineering constraint shaped contrastive learning for years.

The loss, with numbers

Three pairs, in 3 dimensions, so every number can be checked by hand. The texts are deliberately not unit vectors, because the encoders' outputs are not either. The three images are already unit vectors along the axes, so here the image matrix \(I\) happens to equal the \(3 \times 3\) identity:

1. Normalize. The rows of \(T_{\text{raw}}\) have norms \(0.906\), \(0.812\) and \(0.762\). Dividing each row by its norm:

   \[ T = \begin{bmatrix} 0.994 & 0.110 & 0 \\ 0.123 & 0.985 & 0.123 \\ 0 & 0.394 & 0.919 \end{bmatrix} \]

2. Similarities. \(S = I\,T^\top\). Row \(i\) is image \(i\) against the three captions:

   \[ S = \begin{bmatrix} 0.994 & 0.123 & 0 \\ 0.110 & 0.985 & 0.394 \\ 0 & 0.123 & 0.919 \end{bmatrix} \]

   The diagonal is largest in every row and every column: all six retrievals are already right.

3. Logits. Divide by \(\tau\):

   \[ S/\tau = \begin{bmatrix} 14.20 & 1.76 & 0 \\ 1.58 & 14.07 & 5.63 \\ 0 & 1.76 & 13.13 \end{bmatrix} \]

4. Image → text: softmax along each row. Row 2, the one with a real competitor: \(p_{2 \to 2}\), the probability image 2 gives caption 2, is $ e^{14.07} / (e^{1.58} + e^{14.07} + e^{5.63}) = 0.99978$. The three diagonal probabilities are \(0.999995\), \(0.99978\) and \(0.999987\), so

   \[ \mathcal{L}_{I \to T} = -\tfrac{1}{3}\big(\log 0.999995 + \log 0.99978 + \log 0.999987\big) = 7.9 \times 10^{-5} \]

5. Text → image: softmax down each column. Column 3 is now the hard one: caption 3 has cosine \(0.394\) with image 2, so \(p_{3 \to 3} = e^{13.13} / (e^{0} + e^{5.63} + e^{13.13}) = 0.99945\), and \(\mathcal{L}_{T \to I} = 1.9 \times 10^{-4}\). The two directions are not the same number — that is why the loss averages them:

   \[ \mathcal{L} = \tfrac{1}{2}\big(7.9 \times 10^{-5} + 1.9 \times 10^{-4}\big) = 1.3 \times 10^{-4} \]

6. Same embeddings, \(\tau = 1\). The logits are now just \(S\), the softmax is almost flat (row 2 gives \(0.21, 0.51, 0.28\)), and \(\mathcal{L} = 0.626\) — more than half of the \(\log 3 = 1.10\) a blind guess pays. The ranking, and therefore the accuracy, is identical. The temperature decides how much the loss insists, not what the model knows.

Real CLIP does exactly this with \(N = 32{,}768\) and \(d\) = 512 to 1024, and the gradient of \(\mathcal{L}\) flows back through both projections into both encoders.

The same computation, as the lab writes it — clip_loss, line by line. Step through it with the example above (it is the default case), then change a number, the temperature or the case and watch which line moves:

SigLIP: what replaced it

Switch the panel's objective to SigLIP⁵ and the softmax disappears. Each of the \(N^2\) cells becomes an independent binary question — do these two match? — under a plain sigmoid loss with a learned scale \(t\) (it plays the role of \(1/\tau\); a scalar, not the text matrix \(T\)) and bias \(b\):

The bias starts very negative (\(b = -10\)) because \(N - 1\) of every \(N\) cells are negatives; without it, the first steps would be spent only pushing everything apart. Nothing is normalized across the batch: no all-gather (collecting every accelerator's embeddings so each can normalize over the whole batch), no sharded softmax. In the paper, the sigmoid loss beats the softmax clearly at small batches and matches it at large ones, and the gains stop around a batch of 32k for both. Most recent open vision-language models (VLMs: language models that also read images), such as PaliGemma, Idefics2 and 3 and Gemma 3, use a SigLIP-family image encoder, so if you are picking an image tower today, start there — not with the original CLIP.

Zero-shot classification, and what it really is

Write one sentence per class — "a photo of a {class}" — encode them, encode the image, take the highest cosine similarity. No training, no labelled examples, and any set of classes you can describe in words.

Written as a layer, it is a linear classifier whose weights were typed instead of learned. Stack the \(C\) normalized prompt embeddings into a matrix \(W \in \mathbb{R}^{C \times d}\); then for an image embedding \(x\),

where \(w_c\) is the row of \(W\) for class \(c\) and \(\langle \cdot, \cdot \rangle\) the dot product. That is an nn.Linear(d, C, bias=False) with \(W\) as its weight.

To apply CLIP to a new task, all we need to do is "tell" CLIP's text-encoder the names of the task's visual concepts, and it will output a linear classifier of CLIP's visual representations. The accuracy of this classifier is often competitive with fully supervised models.¹

The lab measures it on 1,000 CIFAR-10 test images with ViT-B/32: 89.5% accuracy with "a photo of a {}.", against 87.4% with the bare class name. A linear probe (a logistic regression trained on the CLIP image embeddings of \(k\) labelled CIFAR-10 training images per class) needs 8 to 16 per class to catch up.

The shared space is not evenly shared: the modality gap

All embeddings live on the same unit sphere, but not in the same region of it. Image embeddings crowd into one narrow cone and text embeddings into another⁸. In the lab, two random images have cosine 0.755 on average and two random prompts 0.671, while an image and its own correct prompt reach only 0.258. A linear classifier separates "is this vector an image or a caption?" with 100% accuracy.

This does not break zero-shot classification: the argmax only compares one image against several texts, so a shift common to all texts cancels. In the lab, removing both means moves the accuracy only from 89.5% to 91.3%: subtracting the text mean \(\mu_t\) shifts every class score by the same \(\langle \mu_t, x \rangle\), which cancels in the argmax, while subtracting the image mean \(\mu_i\) shifts class \(c\) by \(\langle w_c, \mu_i \rangle\), which differs per class (and so does the new normalization). Those two effects are the 1.8 points. But the gap explains two things you will see in practice: CLIP similarities between an image and a text are small numbers (0.2 to 0.35 for a good match), and an image is usually closer to other images than to any caption, which matters when you mix both in one retrieval index.

The honest limitations

Where CLIP actually sits in this course

This chapter is here because text-conditioned generation needs a text encoder, and CLIP is where one came from.

Lab: one space, measured

The model's behaviour described on this page is measured in a Colab notebook with the public openai/clip-vit-base-patch32 on CIFAR-10 and CIFAR-100 test images. It runs in the browser; a T4 GPU (free in Colab) makes it take a few minutes, and it also runs on CPU, more slowly. There is nothing to install. Nine sections, each ending in a checkpoint with a check value:

1. the two towers: shapes, widths, parameters and where the projections sit;
2. the contrastive loss computed by hand on a batch, checked against the model's own;
3. the temperature: the learned value, and what it changes and does not change;
4. the batch size as part of the objective, from 2 to 100 candidates;
5. zero-shot classification as a linear layer whose weights are sentences;
6. prompt templates, from the bare class name to an ensemble;
7. image–text retrieval in both directions;
8. the geometry of the space: cosine similarities, the modality gap, a bag of words;
9. zero-shot against a few-shot linear probe — and your own prompts, in a table.

Open the lab in Colab — save a copy to your Drive (File → Save a copy in Drive) to run and edit it.

Key takeaways

1. CLIP trains two encoders to put an image and its caption in the same space. That shared space is an interface between vision and language, not a generative model.
2. The loss is contrastive: the diagonal of the batch similarity matrix is right, everything else is wrong, in both directions. Discrimination, not reconstruction.
3. Temperature decides how much the loss cares about the hardest negative, and CLIP learns it; it changes the loss, not the ranking. Batch size is part of the objective — every other caption is a negative.
4. SigLIP replaced it: an independent sigmoid per pair, no batch-wide normalization, better at small batches and as good at large ones. Use a SigLIP-family encoder today.
5. Zero-shot classification is a linear classifier whose weights are encoded sentences — retrieval with a language index. Impressive, and not magic; the wording of the prompt matters.
6. The shared space has a modality gap: images and texts sit in separate cones, and a correct image–text pair has a smaller cosine than two unrelated images.
7. CLIP is close to a bag of words: counting, spatial relations and compositionality are weak, and that flows through to every model conditioned on it.
8. In this course CLIP is the bridge: it is what lets chapter 20 onward condition on a sentence. And data curation mattered more than the architecture — that is the durable lesson.

Additional

L2-normalized embeddings

An L2-normalized vector has Euclidean length 1. You get it by dividing each component by the vector's L2 norm³:

Why CLIP uses it

• Only the direction is compared. After normalization the dot product is the cosine similarity⁴, a number in \([-1, 1]\), and a long vector cannot win a comparison just by being long.
• Distance and similarity become the same ranking. For unit vectors, \(\|a - b\|^2 = 2 - 2\cos(a, b)\), so nearest-neighbour search by distance and by cosine return the same items.
• The temperature has a fixed scale to act on. Without normalization the logits could grow just by growing the norms; with it, the logits live in \([-1/\tau, 1/\tau]\) and only \(\tau\) sets how sharp the softmax is.

The same constraint — embeddings on a hypersphere — is common wherever similarity is the output: face recognition, retrieval and recommendation.