Artificial Neural Networks and Deep Learning · Insper

Joint Embedding Predictive Architectures

Predict the representation, not the pixels

What should the network predict?

No labels. A network sees part of an image (or of a video) and has to learn from the rest.
What should the loss compare?

  • The data itself — pixels, noise, the next token: the VAE, diffusion, autoregressive models, MAE.
  • Nothing: only who goes with whom — CLIP pulls matching pairs together and pushes the rest apart.
  • A third answer — predict the representation of the hidden part. That is a JEPA.

Three families, three places for the loss

Three ways to learn from pairs (x, y). Generative, like MAE: an encoder and a decoder turn x into a guess of y in pixels and the energy compares pixels. Joint embedding, like SimCLR, VICReg or CLIP: two encoders map x and y to vectors and the energy compares the vectors, with negatives or a regularizer against collapse. JEPA: the encoder of x feeds a predictor that guesses the representation of y, produced by a target encoder that gets no gradient; the energy compares representations. GENERATIVE · MAE D x y the energy compares pixels: every detail of y must be predicted encoder decoder ŷ pixels pixels JOINT EMBEDDING · SimCLR, VICReg, CLIP D x y the energy compares two embeddings: invariance between two views encoder encoder sx sy + negatives or a regularizer JEPA · I-JEPA, V-JEPA D x y the energy compares representations: only what is predictable from x encoder sx predictor z ŝy target enc. EMA · no gradient sy

: the part you see · : the part to predict · : their representations (vectors) · : a distance · : a latent for what does not determine.

The JEPA energy

  • Energy: a number, low when fits , high otherwise (LeCun's energy-based models, 2006).
  • The two encoders need not share weights or architecture.
  • = squared L2 in I-JEPA, L1 in V-JEPA. = every trainable weight.
  • : the car at a fork turns left or right — alone does not decide.

The energy is the prediction error in representation space. No pixel appears in it.

LeCun (2022), A Path Towards Autonomous Machine Intelligence, §4.4.

Why not predict the pixels?

"it is essentially impossible to predict every pixel value of every future frame" — LeCun (2022)

  • The texture of a carpet, the leaves in the wind, the ripples on a pond: unpredictable and irrelevant.
  • A generative model must spend capacity on them, or hide them in its latent variable.
  • A JEPA lets drop them: keeps only what can predict.

LeCun even classes masked and denoising autoencoders as contrastive methods: the corrupted input plays the wrong pair.

The flat energy

Both encoders output the same constant vector for every input. What is the loss? Has the network learned anything?

  • is the same for every → perfectly predictable → for every pair.
  • The energy surface is flat: low on data, low everywhere else. That is collapse.
  • LeCun's four criteria: (1) informative about ; (2) informative about ; (3) predictable from ; (4) carrying little.
  • Criterion 3 alone is satisfied by collapse. 1, 2 and 4 are what prevent it.

Simulator · collapse, live

Four ways out

family methods what prevents collapse
contrastive SimCLR, MoCo, CLIP negatives: the rest of the batch (4,096) or a queue (65,536)
distillation BYOL, SimSiam, DINO, I-JEPA, V-JEPA asymmetry: predictor + stop-gradient + EMA target
information maximization Barlow Twins, VICReg a term: variance up, dimensions decorrelated
distribution matching LeJEPA (SIGReg) embeddings pushed toward an isotropic Gaussian

Stop-gradient: the target branch is treated as a constant — no gradient flows into it. EMA: exponential moving average of weights.

LeCun's split: contrastive (push wrong pairs up) vs regularized (limit the volume of low energy). He bets on the second.

The ablations are blunt

what was removed method ImageNet linear probe
nothing SimSiam 67.7%
the stop-gradient SimSiam 0.1% (chance)
the predictor SimSiam 0.1%
the moving average (: target = online) BYOL 0.3%
the momentum () MoCo training fails
  • Without the stop-gradient, SimSiam "quickly finds a degenerated solution and reaches the minimum possible loss".
  • Nothing in these losses forbids collapse: the dynamics avoid it.

The EMA target encoder, with numbers

: weights of the online (context) encoder, trained by gradient · : weights of the target encoder, never trained by gradient · : momentum.

  • (BYOL, DINO, I-JEPA): each step moves the target of the way.
  • A weight from 250 steps ago still counts : the target is an average over a few hundred versions.
  • A goal that moves too slowly to be chased into the trivial solution.
  • I-JEPA raises linearly from 0.996 to 1: by the end the target is frozen.

Exercise · VICReg by hand

embeddings (rows), dimensions (columns), , .

  1. variance of each column (divide by )
  2. the hinge of each column, and
  3. the covariance , and
  4. which dimension is collapsing?

VICReg by hand · answer

quantity column 1 column 2
variance
std
hinge
  • Column 2 is almost collapsed: the hinge pushes its std up toward .
  • Full loss: invariance (on both views). "Contrastive over components, not over vectors."

How to tell that it collapsed

Std of the normalized embeddings (SimSiam)

  • L2-normalize each embedding, take the std of each dimension over the batch.
  • Spread like an isotropic Gaussian: . Complete collapse: 0.
  • We report std : healthy .

Effective rank (RankMe)

  • → → entropy →
  • Detects dimensional collapse; blind to collapse to one point (scale-invariant).

: singular values of the centred batch of embeddings (n × d).

I-JEPA: four parts

  1. Masks on the patch grid: target blocks (scale 0.15–0.2, aspect 0.75–1.5) and one context block (0.85–1.0), minus every target patch → about 25% of the patches left.
  2. Context encoder : a ViT that sees only the context patches.
  3. Target encoder : the EMA of , sees the whole image; targets cut from its output.
  4. Predictor : a narrow ViT (width 384) — context tokens + one mask token per target position.

: positions of block · : predictor output at · : target encoder output at . Gradient reaches and only.

Simulator · the multi-block mask

The masks are worth tens of points

1% of ImageNet labels, linear probe, ViT-B/16 (Assran et al., 2023, Tables 6–11):

change from the default top-1
default: 4 targets of 0.15–0.2, context 0.85–1.0 54.2
1 target instead of 4 9.0
targets of 0.075–0.2 (some tiny) 19.2
context of 0.40–1.0 31.2
random patches instead of blocks 17.6
ViT-H/16: targets from the encoder's input instead of its output 56.1 (vs 67.3)

Targets large enough to be semantic; context spread enough to be informative.

Simulator · one training step, line by line

x · 6 patches (rows)
W · context encoder θ
W̄ · target encoder θ̄
pos · predictor φ

The step, with numbers

6 patches of 2 numbers, embeddings of 3, context , targets .

line result
context encoder, patch 0
target encoder, patch 4 → layer norm →
predictor, target 4 context mean + →
loss mean of 6 squared errors = 0.6162
one step, lr = 0.5 loss 0.2214; = None
collapsed case (all zeros) loss 0, every gradient 0 — a perfect, stable minimum

Layer norm: subtract the mean of the vector, divide by its std. [0, 2.2, −2]: mean 0.067, std 1.715.

What I-JEPA bought

ImageNet-1K, linear probe on frozen features (I-JEPA paper, Table 1):

method model epochs top-1
MAE ViT-H/14 1,600 77.2
data2vec ViT-L/16 1,600 77.3
I-JEPA ViT-H/14 300 79.3
I-JEPA ViT-H/16, 448 px 300 81.1
iBOT (with view augmentations) ViT-L/16 250 81.0
  • ViT-H/14 in < 72 h on 16 A100 — over 10× less compute than MAE's ViT-H/14.
  • Same ViT-L, 1% labels: pixel targets 40.7% (800 ep.) vs representation targets 66.9% (500 ep.).

Not a uniform win: iNat18 47.6 vs iBOT 57.3; Clevr counting MAE 90.5 vs 86.7; fine-tuned 87.1 vs MAE 87.8.

Why representations — and when not

  • Littwin et al. (2024), deep linear networks: a JEPA learns first the high-influence features (those that predict the target well); MAE's order is set by variance.
  • In natural images most pixel variance is texture and lighting; the semantics have less variance and are more predictable.
  • The bias "takes hold only when the encoder is deep".

So at small scale — a 4-layer ViT, 32 × 32 images, minutes of training — which one do you expect to probe better, a JEPA or an MAE? The lab measures it.

Video: V-JEPA and V-JEPA 2

V-JEPA (2024)

  • 16 frames, tubelets 2 × 16 × 16, ~90% masked, L1 loss
  • ~2 M videos; frozen ViT-H/16: Kinetics-400 82.0, SSv2 71.4
  • image models on SSv2 (motion): DINOv2 50.6, I-JEPA 50.0
  • features vs pixels, same model: K400 73.7 vs 68.6

V-JEPA 2 (2025)

  • 1 M hours of video + 1 M images

  • ViT-g, 1 B parameters, 3-D RoPE
  • SSv2 77.3; Epic-Kitchens anticipation R@5 39.7 (previous best 27.6)
  • with an 8 B LLM: PerceptionTest 84.0

SSv2 = Something-Something v2: actions that only motion distinguishes ("pushing something left" vs "right"). Frozen = encoder not fine-tuned; an attentive probe on top.

V-JEPA 2-AC: the energy becomes a planning cost

  • Freeze the encoder. Train an action-conditioned predictor (~300 M) on < 62 h of unlabeled robot video (Droid): (frame representation, arm state, action) → next frame representation.
  • That predictor is a world model. Plan:

Goal . Three actions predict , , . Which one?

  • L1: · · → the second.
  • Search: cross-entropy method — 800 samples, keep the 10 best, refit, × 10; execute the first action; replan.

On a real arm

Zero-shot, Franka arms in two labs, 10 trials per task per lab (success %, averaged):

reach grasp cup grasp box pick-and-place cup pick-and-place box
Octo (policy fine-tuned on Droid) 100 15 0 15 10
V-JEPA 2-AC 100 65 25 80 65
  • 16 s per action, against 4 min for Cosmos (a video-generation world model), same GPU.
  • No reward, no demonstration of the task: only a goal image.

Ten trials, short horizons (one step, with sub-goals), goals as images. A proof that representation-space planning works — not a general-purpose robot.

LeJEPA: one regularizer instead of the tricks

  • Balestriero & LeCun (2025): the embedding distribution that minimizes downstream risk is an isotropic Gaussian.
  • SIGReg: project the batch on random directions ; test each 1-D projection against (Epps–Pulley):

: empirical characteristic function of the projection; : that of a standard Gaussian. 17 points on [−5, 5], 256 directions.

  • Loss , one hyperparameter, .
  • No stop-gradient, no EMA teacher, no schedulers; ~50 lines; ViT-H/14 79% linear on ImageNet.

MAE, CLIP, JEPA

MAE CLIP I-JEPA
input 1 image, 75% hidden at random image + caption 1 image: context + 4 target blocks
target pixels of the hidden patches which caption is whose representations of the hidden patches
loss lives in input space embedding space embedding space
against collapse not needed negatives stop-grad, predictor, EMA
needs a pixel decoder 400 M pairs, big batches images only
strong at fine-tuning, low-level detail zero-shot, open vocabulary frozen features per unit of compute

Measured in the lab

CIFAR-10, 4-block ViT (0.8 M parameters). Spread: std of normalized embeddings × (healthy ≈ 1).

run loss spread eff. rank linear probe
raw pixels · untrained encoder — — · 0.386 — · 21.4 0.358 · 0.371
I-JEPA, 8 epochs 0.31 0.803 51.3 0.454
no stop-gradient, 1 epoch 0.0009 0.009 39.6 0.403
no EMA (), 1 epoch 0.0098 0.059 13.5 0.338
no EMA, no stop-grad, + VC terms, 4 epochs 1.87 0.832 106.7 0.406
MAE, 8 epochs 0.55 0.531 51.3 0.496
  • The lowest losses belong to the collapsed runs; the rank misses the one that only shrank.
  • No collapse ≠ good features (VC terms). At this scale MAE wins, as slide 20 predicted.

Losses of different objectives are not comparable across rows; compare within a row type.

Toward world models

  • LeCun's 2022 agent: perception, world model, cost, actor, memory, configurator — it plans by running the world model forward: model-predictive control with a learned model and a learned cost.
  • The JEPA was proposed as the world model; H-JEPA stacks levels: near and detailed below, far and abstract above.
  • V-JEPA 2-AC is one working loop. Its weak point: the cost is an L1 distance to a goal image — two states that look alike can be many actions apart.

Next class: world models. Then: TDWM (2026), where the similarity of two frames falls with the number of steps between them, so the latent distance measures time, not looks.

Lab · a JEPA, built and measured

Colab notebook, ~2 h, CIFAR-10, free T4 GPU (~18 min of training on the M2 it was checked on). Each section ends in a checkpoint.

  1. the multi-block mask
  2. the three networks
  3. one step by hand: no gradient into the target
  4. measuring a representation
  5. I-JEPA, trained
  1. collapse on purpose: no stop-grad, no EMA
  2. the regularized route (variance + covariance)
  3. pixels instead: a small MAE, same budget
  4. open task

open in Colab →

Takeaways

  1. A JEPA predicts the representation of the hidden part; the energy is the prediction error in representation space.
  2. Joint embeddings can collapse; a zero loss can mean nothing was learned. Measure std and effective rank.
  3. Four ways out: negatives, stop-grad + predictor + EMA, variance–covariance, SIGReg.
  4. I-JEPA: targets cut from the output of an EMA encoder; large target blocks; the masks are worth tens of points.
  5. Its gain is frozen features per unit of compute, not a uniform win — and at small scale pixels can win.
  6. V-JEPA 2-AC turns the energy into a planning cost on a real robot.
  7. The JEPA was proposed as a world model: the next class.

Outline: (1) what self-supervised learning should predict; (2) the energy view and collapse; (3) four ways out of collapse; (4) I-JEPA, line by line; (5) video and planning with V-JEPA 2; (6) LeJEPA; (7) the bridge to world models. Closes with the Colab lab.

Ask for the first two answers before revealing: the course has seen both. The third is the class.

Point at where D sits in each panel. Left: pixels. Middle and right: vectors. The only difference between the middle and the right is the predictor, and what it buys: a task, not just agreement.

Start with "nothing": the loss goes to 0 and the points to the origin. Then "stop-gradient only": slower, same end. Then EMA + predictor: survives. Then VICReg. Finally raise the noise with EMA and watch the effective rank drop toward 1 — dimensional collapse.

Default first: count the context patches. Then shrink the targets (scale slider to 0.075): the task becomes local. Then 1 target. Then the random strategy (MAE) for contrast.

Walk the 10 lines with the page example. At line 4 ask: why does the target encoder see the whole image? At line 8, point at "None" for W-bar. Then pick the collapsed case and go back to line 7: loss 0, gradients 0.