19. CLIP

Every generative model in the previous chapters could produce an image. None of them could produce the image you asked for, because none of them had any idea what a sentence means. CLIP is where that changes, and it is why this chapter sits in the middle of a module about generation despite not being a generative model at all.

The idea² is almost aggressively simple. Take 400 million (image, caption) pairs off the internet. Train an image encoder and a text encoder — separately, with no shared weights — under one instruction: put an image and its own caption close together, and everything else far apart.

That is the whole objective. What comes out of it is a single vector space that images and sentences both live in, and therefore an interface: you can now ask questions across the modality boundary. "How close is this picture to the phrase a photo of a dog?" becomes a dot product.

The components

• Image encoder — a ViT, or in the original paper a modified ResNet, producing a fixed-size embedding.
• Text encoder — a Transformer, producing an embedding of the same dimensionality. That equality is the entire architectural commitment.
• A projection and an L2 normalization on each side, so every embedding sits on the unit sphere and cosine similarity is just a dot product.
• A contrastive loss (symmetric InfoNCE) over a batch: the diagonal of the image×text similarity matrix is right, everything else is wrong.

The loss, and the two design decisions inside it

Temperature \(\tau\) scales the similarities before the softmax. Small \(\tau\) makes the distribution sharp, so the loss concentrates on the hardest negative; large \(\tau\) treats all negatives alike and the representation stops sharpening. CLIP learns \(\tau\) rather than fixing it, so the model anneals its own difficulty. Drag it in the panel: notice that retrieval accuracy does not change — \(\tau\) affects the loss, not the ranking.

Batch size is part of the objective. Every other caption in the batch is a negative, so a batch of 32,768 gives each example 32,767 negatives. This is not a training-speed decision; it is what determines how hard the task is. It also meant the similarity matrix had to be sharded across hundreds of accelerators, and that engineering constraint shaped contrastive learning for years.

SigLIP: what replaced it

Switch the panel's objective to SigLIP⁵ and the softmax disappears. Each of the \(N^2\) cells becomes an independent binary question — do these two match? — under a plain sigmoid loss with a learned bias:

Nothing is normalized across the batch. No all-gather, no sharded softmax, small batches work, and it trains better at every scale. Every recent vision-language model uses a SigLIP-family encoder, so if you are picking an image tower today, this is the one — not CLIP.

Zero-shot classification, and what it really is

Write one sentence per class — "a photo of a {class}" — encode them, encode the image, take the highest cosine similarity. No training, no labelled examples, and any set of classes you can describe in words.

To apply CLIP to a new task, all we need to do is "tell" CLIP's text-encoder the names of the task's visual concepts, and it will output a linear classifier of CLIP's visual representations. The accuracy of this classifier is often competitive with fully supervised models.¹

The honest limitations

Where CLIP actually sits in this course

This chapter is here because text-conditioned generation needs a text encoder, and CLIP is where one came from.

Key takeaways

1. CLIP trains two encoders to put an image and its caption in the same space. That shared space is an interface between vision and language, not a generative model.
2. The loss is contrastive: the diagonal of the batch similarity matrix is right, everything else is wrong. Discrimination, not reconstruction.
3. Temperature decides how much the loss cares about the hardest negative, and CLIP learns it. Batch size is part of the objective — every other caption is a negative.
4. SigLIP replaced it: an independent sigmoid per pair, no batch-wide normalization, better at every scale. Use a SigLIP-family encoder today.
5. Zero-shot classification is retrieval with a language index. Impressive, and not magic.
6. CLIP is close to a bag of words: counting, spatial relations and compositionality are weak, and that flows through to every model conditioned on it.
7. In this course CLIP is the bridge: it is what lets chapter 20 onward condition on a sentence.
8. Data curation mattered more than the architecture. That is the durable lesson.

Numerical Simulation of CLIP's Contrastive Loss

To illustrate how CLIP works numerically, let's simulate a tiny batch with 3 image-text pairs. We'll assume pre-computed embeddings (in practice, these come from the encoders). Each embedding is a 3D vector for simplicity (real CLIP uses higher dimensions like 512).

Setup:

• Image embeddings (I):

  \( I_1 = [1.0, 0.0, 0.0] \) (e.g., for "cat")
  \( I_2 = [0.0, 1.0, 0.0] \) (e.g., for "dog")
  \( I_3 = [0.0, 0.0, 1.0] \) (e.g., for "bird")

• Text embeddings (T):

  \( T_1 = [0.9, 0.1, 0.0] \) (close to \(I_1\))
  \( T_2 = [0.1, 0.8, 0.1] \) (close to \(I_2\))
  \( T_3 = [0.0, 0.3, 0.7] \) (close to \(I_3\))

• Batch size (\(N\)): \(3\)

• Temperature (\(\tau\)): \(0.07\) (a hyperparameter to scale logits; common in CLIP).

Step-by-Step Calculation:

1. Normalize Embeddings:

   CLIP uses L2-normalized embeddings for cosine similarity. Here, they're already unit-length for simplicity (assume they are).

2. Compute Similarity Matrix (Logits):

   Similarity = \( \displaystyle \frac{(I \cdot T)}{\tau} \) (dot product scaled by τ).

   Calculations:

   \( \begin{align*} \text{Logits}_{I \to T} &= \begin{bmatrix} \text{sim}(I_1, T_1) & \text{sim}(I_1, T_2) & \text{sim}(I_1, T_3) \\ \text{sim}(I_2, T_1) & \text{sim}(I_2, T_2) & \text{sim}(I_2, T_3) \\ \text{sim}(I_3, T_1) & \text{sim}(I_3, T_2) & \text{sim}(I_3, T_3) \end{bmatrix} \\ &= \begin{bmatrix} \frac{1 \cdot 0.9 + 0 \cdot 0.1 + 0 \cdot 0.0}{0.07} & \frac{1 \cdot 0.1 + 0 \cdot 0.8 + 0 \cdot 0.1}{0.07} & \frac{1 \cdot 0.0 + 0 \cdot 0.3 + 0 \cdot 0.7}{0.07} \\ \frac{0 \cdot 0.9 + 1 \cdot 0.1 + 0 \cdot 0.0}{0.07} & \frac{0 \cdot 0.1 + 1 \cdot 0.8 + 0 \cdot 0.1}{0.07} & \frac{0 \cdot 0.0 + 1 \cdot 0.3 + 0 \cdot 0.7}{0.07} \\ \frac{0 \cdot 0.9 + 0 \cdot 0.1 + 1 \cdot 0.0}{0.07} & \frac{0 \cdot 0.1 + 0 \cdot 0.8 + 1 \cdot 0.1}{0.07} & \frac{0 \cdot 0.0 + 0 \cdot 0.3 + 1 \cdot 0.7}{0.07} \end{bmatrix} \\ &\approx \begin{bmatrix} 12.857 & 1.4286 & 0 \\ 1.4286 & 11.4286 & 4.2857 \\ 0 & 1.4286 & 10 \end{bmatrix} \end{align*} \)

   Full image-to-text logit matrix:

   \( \text{Logits}_{I \to T} \approx \begin{bmatrix} 12.857 & 1.4286 & 0 \\ 1.4286 & 11.4286 & 4.2857 \\ 0 & 1.4286 & 10 \end{bmatrix} \)

   CLIP averages both directions, text-to-image logits are the transpose:

   \[ \text{Logits}_{T \to I} = \text{Logits}_{I \to T}^T \]

3. Softmax for Probabilities:

   For each row (image), softmax over logits to get probabilities of matching texts.

   \( \displaystyle \text{Softmax}(I) = \frac{e^{I_i}}{\sum_{j} e^{I_j}} \)

   Calculating exponentials and normalizing:

   \( \begin{align*} \sum_{j} e^{I_j} &\approx \begin{bmatrix} e^{12.857} + e^{1.4286} + e^{0} \\ e^{1.4286} + e^{11.4286} + e^{4.2857} \\ e^{0} + e^{1.4286} + e^{10} \end{bmatrix} \\ &\approx \begin{bmatrix} 383523 \\ 91987 \\ 22031 \end{bmatrix} \end{align*} \)

   Then:

   \( \begin{align*} \text{Softmax}(I) &\approx \begin{bmatrix} \frac{e^{12.857}}{383523} & \frac{e^{1.4286}}{383523} & \frac{e^{0}}{383523} \\ \frac{e^{1.4286}}{91987} & \frac{e^{11.4286}}{91987} & \frac{e^{4.2857}}{91987} \\ \frac{e^{0}}{22031} & \frac{e^{1.4286}}{22031} & \frac{e^{10}}{22031} \end{bmatrix} \\ &\approx \begin{bmatrix} 0.9999 & 0 & 0 \\ 0 & 0.9992 & 0.0008 \\ 0 & 0.0002 & 0.9998 \end{bmatrix} \end{align*} \)

   The diagonal should have high probs.

4. Contrastive Loss:

   Negative log-likelihood of correct labels (diagonal).

   \[ \mathcal{L}_{I \to T} = -\frac{1}{N} \sum_{i=1}^{N} \log(p_{i \to t}) \]

   For this batch:

   \( \mathcal{L}_{I_1 \to T_1} = \log(0.9999) \approx -0.0000 \)

   \( \mathcal{L}_{I_2 \to T_2} = \log(0.9992) \approx -0.0004 \)

   \( \mathcal{L}_{I_3 \to T_3} = \log(0.9998) \approx -0.0001 \)

   \(\mathcal{L}_{I \to T} \approx 0.00016\) (very low loss since embeddings are well-aligned).

   CLIP computes symmetric loss:

   \[ \displaystyle \mathcal{L} = \frac{1}{2} \left( \mathcal{L}_{I \to T} + \mathcal{L}_{T \to I} \right). \]

   In training, gradients update encoders to minimize this. If embeddings were misaligned (e.g., I1 close to T2), loss would be higher.

Additional

L2-normalized embeddings

L2-normalized embeddings are vectors whose length is scaled to a unit of 1, meaning their L2 norm (Euclidean length) is equal to one. This is achieved by dividing each component of the original vector by its total L2 norm, making it a common method for ensuring consistent magnitude and improving the effectiveness of distance-based similarity measures like cosine similarity⁴.

How it works

1. Calculate the L2 norm:

   For a vector \(v=[v_{1},v_{2},...,v_{n}]\), the L2 norm (\(||v||_{2}\)) is the square root of the sum of the squares of its components: \(||v||_{2}=\sqrt{v_{1}^{2}+v_{2}^{2}+...+v_{n}^{2}}\).

2. Divide each component:

   Each element of the vector is then divided by this calculated L2 norm. The resulting normalized vector, \(v^{\prime }\), is:

   \(v^{\prime }=[\frac{v_{1}}{||v||_{2}},\frac{v_{2}}{||v||_{2}},...,\frac{v_{n}}{||v||_{2}}]\). 

Why it is used

• Focus on direction: It helps models focus on the "direction" of the vector in a high-dimensional space rather than its magnitude, which can be useful when the magnitude doesn't carry meaningful information.
• Improves similarity measures: Normalization is crucial for techniques that rely on cosine similarity. L2-normalized embeddings make the similarity score equal to the dot product, simplifying calculations and comparison.
• Prevents magnitude bias: It ensures that embeddings with large magnitudes don't dominate similarity comparisons, preventing bias from large values.
• Used in model architecture: Some models use L2 normalization as a constraint to keep embeddings on a hypersphere, which can be beneficial for tasks like face recognition or out-of-distribution detection.

When to use it

• When using cosine similarity for tasks like retrieval or recommendation.
• In deep learning models where the magnitude of the weights can grow uncontrollably and affect performance.
• When you want to constrain the representation space to a sphere, as it can lead to more stable training.