Artificial Neural Networks and Deep Learning · Insper

CLIP

One space for images and text

Contrastive Language–Image Pre-training

Start with a request

Your ImageNet classifier knows 1,000 classes. Someone asks for every photo of
"a red bus at night". What do you do?

  • Collect labelled photos of red buses at night, add a class, retrain. For every new request.
  • Or: encode the sentence, encode the images, and return the closest ones. No training.
  • The second option needs one thing: images and sentences in the same vector space.

The idea

Take 400 million (image, caption) pairs off the internet.

Train two encoders — no shared weights — under one instruction:

Put an image and its own caption close together, and everything else far apart.

  • What comes out is an interface: "how close is this picture to a photo of a dog?" becomes a
    dot product.
  • CLIP generates nothing. It sits in the generative module because every text-to-image model after
    it needs a text encoder that knows what pictures look like.

Training: one similarity matrix

How CLIP is trained. A batch of four image and caption pairs. The images go down the image tower (image encoder, projection, L2 norm) and become the rows of a 4 by 4 similarity matrix; the captions go down the text tower and become its columns. The diagonal holds the matching pairs. On the right, a cross-entropy across each row, one down each column, and the loss, their average. image tower text tower a matching pair A BATCH OF N = 4 PAIRS image 1 · “a dog on the grass” image 2 · “a red bus at night” image 3 · “two cats on a sofa” image 4 · “a ship at sea” image 3 × 224 × 224 pixels Image encoder ViT-B/32 · [CLS] of width 768 Projection + L2 norm 768 → 512 · unit row Ii caption at most 77 tokens Text encoder Transformer · width 512 Projection + L2 norm 512 → 512 · unit row Tj T1 T2 T3 T4 I1 I2 I3 I4 rows: image → text softmax(S / τ), right column i columns: text → image softmax(S / τ), right row j ℒ = ½ (ℒI→T + ℒT→I) cross-entropy, both ways

The two towers, ViT-B/32

step image side text side
input token ids (at most)
encoder ViT: patches + [CLS], width 768 Transformer, width 512
pooled [CLS]: end-of-text:
projection , no bias , no bias
L2 norm : :
similarity : cosines

87.5 M parameters for images, 63.2 M for text (counted in the lab). The towers share no weight: they meet only in the loss.

: pairs in the batch · B/32: the Base ViT (12 layers, width 768) on 32 × 32 patches · [CLS]: an extra token whose output summarizes the image · end-of-text: the caption's last token, which plays the same role · 77: the text tower's maximum context.

What the loss does not ask for

It never reconstructs, generates or labels: it only tells matching pairs from non-matching ones. The paper measured the alternatives, to the same zero-shot ImageNet accuracy:

objective relative speed
Transformer predicting the caption word by word 1×
predicting which words occur (bag of words) 3×
contrastive: which caption goes with which image 12× (4× the bag of words)

Discrimination is a much cheaper objective than generation — and nearly as informative.

zero-shot: classifying with no training example of the task, only from the class names written as sentences (slide 17).

The loss

, : the normalized embeddings; row of each comes from pair .

  • Row of is an -way classification: which caption belongs to image ? The answer is .
  • The columns ask the reverse: which image belongs to caption ?
  • Symmetric InfoNCE: a softmax cross-entropy whose labels are arange(N).

: the dimension of the shared space (512 in ViT-B/32) · : the temperature · CE(logits, diag): cross-entropy where row 's right answer is column · arange(N) .

By hand: normalize, then compare

. Row norms of : , , — not 1.

  • Divide each row by its norm. Since the image matrix here equals the identity, : column is caption normalized, e.g. .

  • The diagonal wins every row and every column: all six retrievals are right.

By hand: the two directions

Divide by (e.g. ) to get the logits (the scores the softmax takes) and take the softmax both ways:

direction of the right answer, per pair loss
rows (image → text) · ·
columns (text → image) · ·
  • Row 2: .
  • Column 3: caption 3 has cosine with image 2, so .
  • Not the same number — that is why the loss averages them: .

Same embeddings,

Nothing moves: same vectors, same cosines. Only goes from to .
What happens to the loss? To the accuracy?

  • The logits are now itself and the softmax goes almost flat: row 2, , becomes .
  • (the two directions give and ), more than half of the (natural log) a blind guess pays, giving to each caption.
  • The ranking — and so the accuracy — is identical: 3 out of 3.

The temperature decides how much the loss insists, not what the model knows.

Simulator · the loss code, line by line

I_raw · images
T_raw · captions

Simulator · the batch similarity matrix

The temperature is learned

  • Stored as , initialised at , clipped so that .
  • The released models end at the clip: , . Training kept asking for a sharper softmax.

The lab, one CIFAR-10 image per class with a photo of a {class}.:

1 0.07 0.01 0.003
loss 2.2468 1.5728 0.2698 0.4019

At every : 100% image → text (10 of 10), 90% text → image (9 of 10). Below 0.01 the loss rises again — the caption whose best image is the wrong one costs more the sharper the softmax.

A blind guess over 10 captions pays log 10 = 2.30.

The batch size is part of the objective

Same model, same temperature. The batch grows from 2 to 100 pairs. What happens to the loss and the accuracy?

2 10 20 50 100
0.69 2.30 3.00 3.91 4.61
loss ? ? ? ? ?
accuracy ? ? ? ? ?

CIFAR-100 test images, one per class, with a photo of a {class}. · : what a blind guess pays, giving to each caption · accuracy: fraction of images whose nearest caption, among the batch's , is the right one.

The batch size · answer

2 10 20 50 100
0.69 2.30 3.00 3.91 4.61
loss 0.071 0.469 0.733 1.183 1.592
accuracy 0.978 0.868 0.807 0.710 0.636
chance, 0.50 0.10 0.05 0.02 0.01
  • Nothing in the model changed: the same embeddings face more competitors. The loss goes up, the accuracy down.
  • Each of the other captions is a negative: a bigger batch is a harder exam. CLIP trained at : 32,767 negatives per image.

SigLIP: a sigmoid per pair

  • Each of the cells is an independent "match or not?" — no softmax across the batch.
  • The bias starts at : of every cells are negatives. With , the model starts by saying "no match" to everything.
  • No batch-wide normalization: small batches work, and no all-gather is needed. Clearly better at small batches, as good at large ones.

on the diagonal (), off it · : learned scale (it plays the role of ; a scalar, not the matrix ) · : the sigmoid, · : learned bias · : the cell's cosine · all-gather: collecting every accelerator's embeddings, which the batch-wide softmax requires.

Most recent open VLMs (vision-language models: language models that also read images), such as PaliGemma, Idefics2 and 3 and Gemma 3, use a SigLIP-family image encoder. Picking an image tower today? Start there.

Zero-shot classification

Zero-shot classification with CLIP. Above, the class names become one prompt each, go through the text tower and become the rows of a matrix W, C by d. Below, the image goes through the image tower and becomes a unit vector x. The logits are W x divided by tau; the class with the largest one is the prediction. The whole thing is a linear layer without bias whose weights were typed. C CLASS NAMES catdogship · … ONE PROMPT PER CLASS “a photo of a cat.”“a photo of a dog.”“a photo of a ship.” … Text tower encoder · proj. · L2 W · C × d wcat wdog wship image the one to classify Image tower encoder · projection · L2 x · d = 512 logits = W x / τ cosine × 100 cat0.26 → 26 dog0.22 → 22 ship0.19 → 19 ŷ = cat argmax of the logits THE SAME THING, AS A LAYER nn.Linear(512, C, bias=False) weight = W: typed, not trained

It is a linear layer whose weights you typed

Stack the normalized prompt embeddings (one per class) into ; is the image's normalized embedding:

  • That is nn.Linear(512, C, bias=False) ( in, classes out) with as its weight. Zero trained parameters.
  • An eleventh class? Write an eleventh sentence.
  • Lab, 1,000 CIFAR-10 test images (100 per class, 32 × 32 upscaled to 224 × 224): 0.895 accuracy. Most frequent mistake: deer → horse (17 of the 100 deer).

"Zero-shot" is doing a lot of work

CLIP saw 400 million captioned images. Common objects were described there many times. This is
closer to retrieval from a huge training set with a natural-language index than to learning
from nothing.

  • What is new: the class set is defined at inference time, in language, by whoever is typing.
  • How much is a sentence worth? A linear probe (a logistic regression trained on the CLIP embeddings of labelled training images per class) needs 8 to 16 per class to reach the 0.895 (lab: 0.602 with 1, 0.832 with 4, 0.911 with 16).

The wording of the prompt

Same model, same 1,000 images, no training: only each class's sentence changes. Rank these from worst to best.

prompt CIFAR-10 accuracy
{} (bare class name) ?
a photo of a {}. ?
a blurry photo of a {}. ?
a photo of a {}, a type of flower. ?
average of 18 templates ?

{} takes the class name (cat, ship…) · CIFAR images are blurry at 224 px · average: the mean embedding of the 18 sentences the CLIP authors wrote for CIFAR-10.

The wording of the prompt · answer

prompt CIFAR-10 accuracy vs. bare name
{} (bare class name) 0.874 —
a photo of a {}. 0.895 +0.021
a blurry photo of a {}. 0.894 +0.020
a photo of a {}, a type of flower. 0.893 +0.019
average of 18 templates 0.916 +0.042
  • CLIP saw captions, not labels: a lone word is an unusual caption.
  • The wrong context costs almost nothing: the same words on every class cancel in the argmax.

ImageNet, in the paper: +1.3 points for "a photo of a {}.", +3.5 more for 80 templates. A workaround, not a property.

Retrieval: the same dot product, read the other way

Text → image

  • Rank 1,000 CIFAR-10 images for a photo of a {class}.
  • Precision@10 (fraction of the top 10 that belong to the class), mean over the 10 classes: 0.96
  • Free text works too: a red car, a ship on the open sea.

Image → text

  • Rank the 100 CIFAR-100 captions (one per class) for each of 500 images.
  • Recall@1 (the right one comes first): 0.634
  • Recall@5 (the right one is in the top 5): 0.866

One dot product per candidate: this is how CLIP-based search, CLIPScore and dataset filtering work.

The geometry: a modality gap

In the lab, a cat and a photo of a cat. have cosine 0.21 — and it is a right pair. What is the cosine between two random images?

  • Two random images: 0.755. Two random captions: 0.671 (means over all pairs, almost all unrelated).
  • An image and its own prompt: 0.258; the wrong prompts: 0.201 (mean over the 1,000 CIFAR-10 images).
  • "Image or text?" — a logistic regression gets 100%.

Images sit in one cone of the sphere, texts in another (Liang et al., 2022).

The modality gap, drawn

The modality gap, sketched in two dimensions. Eight images, in purple, sit in one narrow cone of the unit sphere and their eight captions, in green, in another; an image is closer to the other images than to its own caption. With the button, each modality's mean is subtracted and the vectors are normalized again: both spread around the sphere and each image lands next to its own caption. images captions

Why the gap does not break zero-shot

  • The argmax compares one image against several texts: a shift shared by all texts cancels.
  • Lab: subtract each modality's mean (and normalize again) → zero-shot only moves from 0.895 to 0.913.
  • What changes is the scale: right pair 0.258 → 0.241, wrong pairs 0.201 → −0.016.

It does matter when images and texts share one index: an image is usually closer to other
images than to any caption. And it is why "good" CLIP similarities are small numbers (0.2–0.35).

A bag of words

caption A caption B cosine
a dog chasing a cat a cat chasing a dog 0.992
a cup on top of a book a book on top of a cup 0.976
a red cube and a blue sphere a blue cube and a red sphere 0.974
one cat three cats 0.884
a dog chasing a cat dog a cat a chasing 0.893
a dog chasing a cat a photo of an airplane 0.692

The embedding knows which words are there far better than how they are arranged. A shuffled template, photo a {} of a., still classifies at 0.896 (vs. 0.895 for the original).

The honest limitations

  • Counting and spatial relations: "three cats" ≈ "a cat"; the paper lists counting among its weak tasks.
  • Composition: which object has which attribute, who did what to whom (Yuksekgonul et al., 2023).
  • Prompt sensitivity: the wording moves accuracy by points.
  • The internet's associations: documented failures on race, gender and occupation — inherited by every model conditioned on CLIP.

Where CLIP sits in this course

use what CLIP provides
conditioning diffusion SD cross-attends to CLIP text embeddings; SD3 and FLUX add T5 for long, compositional prompts
the image tower of a VLM today usually SigLIP-family
retrieval one dot product per candidate
evaluation CLIPScore — blind to word order, like CLIP
data curation LAION was filtered by CLIP similarity

CLIP is the bridge: from the next class on, every generator is conditioned on a sentence.

SD: Stable Diffusion (next class) · SD3, FLUX: later text-to-image models · T5: the encoder of Google's text-to-text Transformer · VLM: a language model that also reads images · CLIPScore: the CLIP cosine between a generated image and its prompt, rescaled · LAION: an open dataset of billions of web image–text pairs.

The lesson that outlived the architecture

  • DataComp, a benchmark built on the OpenCLIP code, held the architecture and the compute fixed and varied only the data.
  • A filtered fraction of a pool beat the whole pool.
  • DataComp's best filtered set trained a ViT-L/14 (the Large ViT, 14 × 14 patches) past OpenAI's own, at the same compute.

At this scale, dataset construction is the model architecture.

Lab · one space, measured

Colab notebook, a ~2 h activity (a few minutes of compute on a free T4 GPU, 10–20 min on CPU), openai/clip-vit-base-patch32 on CIFAR-10 and CIFAR-100. Each section ends in a checkpoint.

  1. the two towers: shapes and widths
  2. the contrastive loss by hand, against the model's
  3. the temperature
  4. the batch size as part of the objective
  5. zero-shot as a linear layer
  1. prompt templates
  2. retrieval in both directions
  3. the geometry: modality gap, bag of words
  4. zero-shot vs few-shot — and your own prompts, scored on fresh images

open in Colab →

Takeaways

  1. Two encoders, one space: an interface between vision and language, not a generator.
  2. The loss is contrastive and symmetric, : the diagonal is right, in both directions.
  3. Temperature changes the loss, not the ranking. Batch size sets how hard the exam is.
  4. SigLIP: a sigmoid per pair, no batch-wide softmax. The image tower to pick today.
  5. Zero-shot = a linear layer whose weights are sentences; the wording matters.
  6. A modality gap and a bag of words: what the loss never asked for.
  7. Data curation mattered more than the architecture.

Outline: (1) the idea and the two towers; (2) the contrastive loss, by hand; (3) its two design decisions (temperature and batch size) and SigLIP; (4) zero-shot classification and prompts; (5) the geometry of the space and the limitations; (6) where CLIP sits in the course. Closes with the Colab lab.

Let the class answer first. The point: a fixed label set is decided at training time; with CLIP the class set is decided by whoever is typing.

Step through it: the batch, the two towers, the projections into one space, the matrix and the two directions of the loss. Hover a box to see its arrows. Purple is the image side, green the text side; the orange cells are the matching pairs.

The projections are the only reason the dot product is defined: the towers do not even have the same width. After normalization, dot product = cosine.

Start with "the example on the page": the numbers are those of the three previous slides. Step line by line. Then: τ = 1 (loss 0.626, same accuracy), "a hard negative" (text → image accuracy drops to 2/3; image → text stays 3/3) and "blind guess" (loss = log 3).

Start at N = 16, tau = 0.07. Drag tau to 1: the loss jumps, the accuracy does not move. Then batch size 4 → 64 at fixed tau: the same kind of embeddings, a harder exam. Finally switch the objective to SigLIP.

Why so small: the cosines of right and wrong captions differ by a few hundredths; only a large 1/tau turns that into a confident softmax.

Lab, section 4: CIFAR-100 test images, one per class, 200 random batches per N, same model and same tau. At N = 32,768 the similarity matrix had to be sharded over hundreds of accelerators.

Step through it: the prompts become the weights W, the image becomes x, and the prediction is the argmax of W x / τ. The cosines are of the size the lab measures.

Press the button: subtracting each modality's mean opens the two cones, and each image lands next to its own caption. The numbers are the lab's.