Latent Space

A latent space is the space in which a network re-describes its data: a list of numbers per input that nobody labelled, learned because the loss needed it. Almost every chapter of this course builds one. A CNN's last feature map, a token's embedding, the residual stream of a Transformer, CLIP's shared space, a VAE's code, the 64 × 64 × 4 grid Stable Diffusion runs on, a world model's state: all of them are latent spaces, and each one is shaped by a different loss. This page gathers what the chapters say about them, then looks closely at the four chapters that are about nothing else. The embedding page looks at the same spaces from the side of the vectors.

The latent of each chapter

All the techniques on one map

Every technique this page mentions, in five families. Click a box for what its latent is, how big it is and what shapes it, with a link to the chapter. Next walks through the seven numbered links between the VAE, JEPA, world models and TDWM that the last sections of the page take apart.

Why a latent can be small, and why it can be large

Small. Data does not fill the space it is written in. The curse-of-dimensionality page puts the intrinsic dimension of MNIST at around 12 to 15, not 784, and the deep-networks chapter lists the autoencoder as the architecture whose bet is that "the data lies on a lower-dimensional manifold". A straight projection cannot follow a curved one: on the spring of the reduction page, PCA puts 72% of the neighbours in the wrong place and t-SNE 1%. That is the case for learned, nonlinear latents.

Large. High-dimensional space is roomy. The cosine between two random directions has mean zero and standard deviation \(1/\sqrt d\): at \(d = 1000\), 99% of pairs have \(|\cos\theta| < 0.082\) (A4 · Everything becomes orthogonal). Almost orthogonal directions are cheap, which is "the reason a 768-dimensional embedding can carry far more than 768 distinguishable things".

Looked at. A latent can only be seen through a projection, and each projection lies differently. The reduction page's box "Seven things you may not say about a t-SNE or UMAP plot" applies to every plot of an embedding: on uniform noise in 20 dimensions the silhouette is 0.042, and on its t-SNE map it becomes 0.333.

Features: the space a classifier builds

A CNN trades resolution for channels: "the tensor gets smaller while what it describes gets richer" (1 × 28 × 28 → 64 × 7 × 7, then a 64-vector after global average pooling). Whatever the last layer reads is a latent, and the transfer-learning chapter measures it with a linear probe: 1,538 trainable parameters on a frozen DistilBERT, which "can only draw a hyperplane in the space the encoder already built". The probe is the course's standard test of a latent; it comes back for ViT, CLIP, JEPA and TDWM. Frozen features are also an economy: they "can be computed once and cached".

Embeddings: from names to vectors

A token id is a name: "Id 4 (␣climb) is not twice id 2 (␣cat)". The embedding layer turns each name into a learned row of \(E \in \mathbb R^{V \times d}\) — 50,257 rows of 768 numbers in GPT-2, 38.6 million parameters, 31% of the model. "A trained table has no readable columns: what a dimension means is spread over many of them." The row is the same whatever the neighbours, which is the limitation attention fixes: in the "bank" example the embedding gives both senses the same vector and attention returns "one vector per token, now with context".

Inside the Transformer the latent keeps one shape, \(n \times d\), from the first block to the last, and each block only adds a correction to it — the residual stream. At the end, unembedding reads it with one direction per token: column \(j\) "is a direction in the \(d\)-dimensional space that favours token \(j\)". Tying that matrix to the embedding table works "because both speak about the same space". And the LLM chapter explains why the space has structure at all: "compression at this scale forces structure".

Shared spaces: CLIP

CLIP puts images and sentences in "a single vector space", 512 numbers on the unit sphere, and "that shared space is the entire architectural commitment". Zero-shot classification is then "a linear classifier whose weights were typed instead of learned". The space is less shared than it looks — the modality gap: image–image cosine is 0.755, prompt–prompt 0.671 and image–own-prompt only 0.258, and a linear classifier separates images from captions 100% of the time.

The same idea measures generators. FID compares two Gaussians fitted in Inception's 2048-dimensional feature space; BERTScore and MAUVE compare text in an embedding space; CLIPScore compares an image with its prompt in CLIP's — and inherits its blind spots: "a red cube on a blue sphere" and "a blue cube on a red sphere" embed almost identically.

Generative latents

• A code with holes. An autoencoder's latent "is full of holes and you cannot see them"; the VAE fills them with a Gaussian per input and pulls all of them toward \(\mathcal N(0, I)\). "The overlap between those Gaussians is what fills in the space between training examples." With β → 0, roughly 90% of prior samples decode to nothing (β is the whole story).
• A latent with bridges. A GAN maps "one connected latent space onto one connected set, so it has to stretch thin bridges between separate modes": with a 1-D latent the generator covered 2 of 8 modes and 57% of its samples landed off the data. StyleGAN's \(\mathcal W\) space is the one place in the course where directions mean attributes — age, pose, expression — "so editing is vector arithmetic" (the lineage).
• A latent that is a small image. Latent diffusion runs in a VAE's 4 × 64 × 64 grid: 786,432 numbers become 16,384, 48× fewer (\(8 \times 8\) per side, times 3/4 for the channels). "The autoencoder handles perceptual detail… the diffusion model handles semantics and composition." The 4 channels became 16 in SD3 and FLUX (what a VAE is for). Guidance shows what leaving the latent's data manifold costs: 1% of samples off it at \(w = 3\), 17% at \(w = 10\), 63% at \(w = 15\).
• A latent cut into tokens. DiT patchifies the grid: 4 × 32 × 32 with \(p = 2\) is 256 tokens; SD3's 16 × 128 × 128 is 4,096. Video adds time: a 3-D VAE compresses 4× in time and 8 × 8 in space. Flow matching moves particles in the same latent.
• A latent of integers. Autoregressive generation snaps the grid to a codebook: 256 tokens of 14 bits is 3,584 bits, about 450 bytes, for an image of 196,608 numbers. The failure is a codebook that is mostly unused — the original VQGAN used 3.4% of 16,384 codes (training a quantizer) — and MAR drops the quantizer because "everything between codebook entries is unrepresentable".

VAE, JEPA, world models and TDWM up close

Four chapters are about nothing but the latent: what should it keep, and what stops it from keeping nothing? The VAE answers with pixels and a Gaussian prior. The JEPA answers with embeddings and a guard against collapse. World models add the action and use the latent to plan. TDWM lets time decide which frames should look alike. Three of them descend from each other, and the VAE sits at the root of two of the links.

The numbered links 1 to 7 of the map above are the ones this section takes apart.

One skeleton, four objectives

Every model on the map has an encoder, a latent and something trained on top of it. What changes is what the loss compares and what keeps the latent from collapsing to a constant:

Two collapses with one name

Both families have a failure called collapse, and both mean that the code stops depending on the input. They are not the same failure:

The asymmetry is the whole difference between predicting pixels and predicting embeddings. Pixels cannot collapse, so the VAE only collapses when the objective says the code is not worth its price. An embedding target moves with the encoder, so a constant is the global minimum of the bare prediction loss and every JEPA has to rule it out explicitly. The JEPA chapter shows it on CIFAR-10: without the stop-gradient the spread of the embeddings falls from 0.803 to 0.009 (how to tell that a representation collapsed).

The Gaussian, twice: the VAE's KL and SIGReg

The VAE and LeJEPA both push a latent toward \(\mathcal N(0, I)\), for different reasons and with a different bill. Average the VAE's KL over the data and split it in two, with \(q(z) = \mathbb E_x\, q(z \mid x)\) the aggregate posterior (the cloud of all codes together)¹:

It takes one line: inside the expectation, \(\log q(z \mid x) - \log p(z) = \big[\log q(z \mid x) - \log q(z)\big] + \big[\log q(z) - \log p(z)\big]\), and the two brackets average to the two terms.

The VAE pays for both. SIGReg looks only at the batch of embeddings, projected on random directions, so it checks only the second term: information is free. The worked example of the VAE chapter (\(p(z) = \mathcal N(0, 1)\), \(p(x \mid z) = \mathcal N(2z, 1)\)) shows the split exactly. With the β-optimal guess \(q^*(z \mid x) = \mathcal N\big(\tfrac{2x}{4+\beta}, \tfrac{\beta}{4+\beta}\big)\) and \(x \sim \mathcal N(0, 5)\):

At β = 1 the encoder is the true posterior, the codes together are exactly \(\mathcal N(0, 1)\) and the whole 0.805 nats is information about \(x\) (\(\tfrac12 \log 5\), as in step 4 of the VAE lab). SIGReg would charge nothing for that code. This is why a β-VAE trades information for a regular latent while a SIGReg JEPA does not — and why TDWM looks for the information LeWM loses in the prediction term, not in SIGReg.

The main loss decides what the latent keeps

The dashed link of the map is the same lesson learned twice. A regularizer shapes the latent; the main loss decides what goes into it.

• VAE. In section 6 of the VAE lab the decoder gets a hint, the image itself average-pooled, next to \(z\). With β = 1 the KL falls from 15.83 nats (no hint) to 1.82 (7 × 7) and 0.02 (28 × 28): a decoder that can do the job alone makes the code worthless, and the prior becomes the optimum.
• LeWM. Next-embedding MSE is lowest when the future is easy to predict, and the surest way to make it easy is to stop representing what is hard to anticipate. On OGBench Scene a linear probe recovers the drawer from LeWM's embeddings with \(R^2 = 0.95\), but the cube with 0.27 and the arm with 0.39². SIGReg cannot object: a Gaussian cloud can be built from the drawer alone. TDWM's own probe reaches 0.56 on the cube and 0.69 on the arm.

In both cases the fix is in the objective, not in the regularizer: free bits or a weaker decoder for the VAE; for TDWM, a loss in which every frame of the clip has to be told apart from the others.

The geometry is what a planner uses

A world model plans by distances in its latent, so the latent's geometry is the planner's map. In the world-models chapter the latent distance has a rank correlation of 0.93 with the straight-line distance and only 0.41 with the length of the path an agent must actually walk; CEM solves 1 task in 10 with the first cost and 10 in 10 with the second. TDWM's answer is to make the geometry temporal: frames a few steps apart close, distant frames nearly orthogonal (why the geometry matters).

Four ideas that come back in every chapter

1. The loss decides what the latent keeps. Reconstruction keeps every detail, and a VAE blurs what it cannot; a contrastive loss keeps what tells pairs apart, and CLIP reads captions like a bag of words; next-embedding prediction keeps what is predictable, and LeWM forgets the cube.
2. A linear probe is the test. Transfer learning, ViT, CLIP, JEPA and TDWM all judge a latent by what a hyperplane can read from it.
3. Geometry is a design choice. A Gaussian (VAE, SIGReg), a sphere (CLIP, TDWM), a grid with channels (latent diffusion, DiT), a codebook (VQ). The choice decides which distances mean something.
4. Off the manifold, nothing was trained. The autoencoder's holes, the GAN's bridges, guidance pushed too far and a policy that exploits its world model are the same failure: a point the decoder or the predictor never saw.

Reading order for the four

1. VAE — the ELBO, a Gaussian latent and posterior collapse.
2. JEPA — predict the embedding, not the pixels; collapse and the four ways out, SIGReg among them.
3. World models — V–M–C on a VAE, the RSSM's ELBO over time, the latent predictive family and planning with CEM.
4. TDWM — a JEPA world model in which time, not a regularizer, keeps the latent alive.