Joint Embedding Predictive Architectures (JEPA)

Every self-supervised method has to answer one question: when a network learns from unlabeled data, what should it predict? The course has met two answers. Generative models predict the data itself — the pixels of a VAE reconstruction, the noise of a diffusion model, the next token of an autoregressive model. CLIP predicts nothing: it pulls the embeddings of matching pairs together and pushes the others apart.

A Joint Embedding Predictive Architecture gives a third answer: predict the representation of the missing part, not the part itself. Encode what you can see, encode what is hidden with a second encoder, and train a predictor to map the first onto the second. The loss never touches a pixel.

Yann LeCun proposed it in 2022 as the core of an architecture for machines that learn models of the world¹. Since then it has become a line of work with measured results: I-JEPA on images (2023)², V-JEPA on video (2024)⁵, V-JEPA 2 (2025), whose action-conditioned version plans pick-and-place motions on a real robot arm from 62 hours of unlabeled robot video⁶, and LeJEPA (2025), which replaces the bag of tricks that keeps these models from collapsing with one provable regularizer¹³. This chapter builds the idea from the energy-based view, takes the collapse problem apart, follows the recipe from images to video to planning, and ends where the next class starts: world models.

Slides

Click the slides and use ← → to move through them; F goes full screen and P opens the presenter view, with the notes. The simulators of this page run inside the slides. Open the slides in a new tab

Three answers to "what should it predict?"

Take a pair \((x, y)\) that belong together: two crops of the same photo, the visible and the hidden patches of an image, the present and the next second of a video. There are three ways to build a training signal from it, and they differ in where the comparison happens:

• Generative (MAE³, the VAE, diffusion). An encoder reads \(x\), a decoder produces a guess \(\hat y\) in input space, and the loss compares pixels. Everything about \(y\) has to be predicted, including what cannot be predicted from \(x\).
• Joint embedding (SimCLR⁷, VICReg¹⁰, CLIP). Two encoders map \(x\) and \(y\) to vectors \(s_x\) and \(s_y\), and the loss makes them agree. The encoders are free to drop details, but nothing stops them from mapping every input to the same vector — so these methods need negatives or a regularizer.
• JEPA. The encoder of \(x\) feeds a predictor that guesses \(s_y\), the output of a second encoder on \(y\). The comparison happens in representation space, like a joint embedding, but the task is a prediction, like a generative model: the predictor has to know where the hidden part is and what it should contain.

The energy-based view

LeCun's framing is an energy-based model⁴: a scalar function \(E(x, y)\) that is low when \(y\) is compatible with \(x\) and high otherwise. Training shapes the function; inference looks for the \(y\) (or, later, the action) with the lowest energy. A JEPA's energy is the prediction error in representation space¹:

Two features of that formula carry the whole argument.

It does not have to predict every detail. In the paper's words, "it is essentially impossible to predict every pixel value of every future frame"¹ — the texture of a carpet, the leaves of a tree in the wind, the ripples on a pond. A generative model must either spend capacity on those details or push them into its latent variable. A JEPA can let \(\text{Enc}_y\) throw them away, so that \(s_y\) keeps only what \(x\) can predict.

It can collapse. If both encoders output the same constant vector for every input, \(s_y\) is perfectly predictable and \(E = 0\) for every pair: the energy surface is flat, and the model has learned nothing. Every method in the next section is a way of keeping the energy low on data and high elsewhere. LeCun groups them into two families:

• Contrastive: push the energy of real pairs down and of "wrong" pairs up. SimCLR, MoCo and CLIP do this with negatives; LeCun counts denoising and masked autoencoders here too, since a corrupted input is a wrong pair¹. His objection is cost: the number of wrong pairs needed to shape the surface can grow exponentially with the dimension of the representation.
• Regularized (non-contrastive): limit the volume of space that can have low energy, so it can only "shrink-wrap" the data. He argues these are "much more likely to be preferable in the long run"¹.

His four criteria for training a JEPA say what the regularizer has to achieve:

1. maximize the information content of \(s_x\) about \(x\);
2. maximize the information content of \(s_y\) about \(y\);
3. make \(s_y\) easily predictable from \(s_x\);
4. minimize the information content of the latent \(z\).

Criterion 3 alone is satisfied by collapse. Criteria 1, 2 and 4 are what prevent it — and 4 exists because a \(z\) as large as \(s_y\) would let the predictor copy the answer from \(z\) and drive the energy to zero everywhere.

Collapse, and four ways out

Collapse is not a theoretical worry. Here is a two-dimensional joint embedding trained by gradient descent: 64 inputs from 4 classes, two noisy views of each, a linear encoder, and the loss "make the two views agree". Change what keeps the branches apart and watch the loss and the spread of the embeddings:

With nothing in the way, the loss drops to zero and every point lands on the origin. The best loss of the four runs belongs to the network that learned nothing. The four families of methods in the literature differ in what they put in the way:

The ablations that made these mechanisms famous are blunt. SimSiam without the stop-gradient "quickly finds a degenerated solution and reaches the minimum possible loss", and its ImageNet accuracy drops from 67.7% to 0.1%, chance level; without the predictor it also falls to 0.1%¹¹. BYOL with \(\tau = 0\), a target that is an instant copy of the online network (stop-gradient but no moving average), reaches 0.3%⁹.

The EMA target encoder, with numbers

The target encoder's weights \(\bar\theta\) follow the online weights \(\theta\) slowly:

With \(m = 0.996\) — the starting value of BYOL, DINO and I-JEPA — each step moves the target 0.4% of the way. A weight from 250 steps ago still has \(0.996^{250} \approx 0.37\) of its influence, so the target is an average over the last few hundred versions of the network: a stable goal that moves too slowly to chase the online encoder into a trivial solution. I-JEPA raises \(m\) linearly to 1 over training, freezing the target by the end².

Variance and covariance, with numbers

VICReg¹⁰ writes criteria 1 and 2 as two terms on a batch of \(n\) embeddings \(Z \in \mathbb{R}^{n \times d}\):

where \(z^j\) is column \(j\) (one dimension across the batch), \(\gamma = 1\) is the target standard deviation, \(\epsilon = 10^{-4}\) avoids a zero square root, and \(C(Z)\) is the \(d \times d\) covariance matrix. The full loss adds an invariance term (the mean squared distance between the two views) with weights \(\lambda = \mu = 25\) for invariance and variance and \(\nu = 1\) for covariance.

Take \(n = 4\) embeddings in \(d = 2\):

• Variance. Column 1 has variance \(4/3 = 1.333\) (with \(n - 1\) in the denominator), standard deviation \(\sqrt{1.333 + 0.0001} = 1.155\), hinge \(\max(0, 1 - 1.155) = 0\). Column 2 has variance \(0.08/3 = 0.0267\), standard deviation \(0.164\), hinge \(1 - 0.164 = 0.836\). So \(v = (0 + 0.836)/2 = 0.418\): the second dimension is nearly collapsed and the term pushes it out.
• Covariance. \(C_{12} = (1{\cdot}0 + (-1){\cdot}0 + 1{\cdot}0.2 + (-1){\cdot}(-0.2))/3 = 0.133\), so \(c = (0.133^2 + 0.133^2)/2 = 0.0178\): the two dimensions are slightly redundant.

LeCun's own gloss on the difference: VICReg "is contrastive over components, while traditional contrastive methods are contrastive over vectors"¹. It needs no negatives and no large batch.

How to tell that a representation collapsed

Two numbers, both used in the lab, and both computed on a batch of embeddings:

• Standard deviation of the normalized embeddings. SimSiam's diagnostic¹¹: L2-normalize each embedding and take the standard deviation of each dimension across the batch. If the embeddings were spread like an isotropic Gaussian it would be close to \(1/\sqrt{d}\); under complete collapse it is 0. We report it multiplied by \(\sqrt d\), so that "healthy" is near 1.
• Effective rank (RankMe¹⁴). With \(\sigma_k\) the singular values of the centred batch and \(p_k = \sigma_k / \sum_i \sigma_i\), the effective rank is \(\exp\!\big(-\sum_k p_k \log p_k\big)\): \(d\) when every direction is used equally, 1 when one direction carries everything. For singular values \((3, 1, 0)\): \(p = (0.75, 0.25, 0)\), entropy \(0.562\), effective rank \(e^{0.562} = 1.75\) — three dimensions, less than two in use. This detects dimensional collapse¹⁵, the partial kind, where the embeddings stay spread but only along a few directions. It is scale-invariant, so it cannot see complete collapse; the standard deviation can.

I-JEPA: the recipe for images

I-JEPA² is the first JEPA trained at scale. Everything is a Vision Transformer working on patches, and the training step has four parts:

1. Masking. From the patch grid, draw \(M = 4\) target blocks, each covering 15–20% of the image with aspect ratio between 0.75 and 1.5, and one context block covering 85–100%. Remove from the context every patch that falls in a target. The code then cuts every image of a batch to the smallest context in it, so that on average the encoder sees about a quarter of the patches (the paper's Table 6).
2. Context encoder \(f_\theta\), a ViT that sees only the context patches.
3. Target encoder \(f_{\bar\theta}\), the EMA of the context encoder, which sees the whole image. The targets \(s_y\) are cut from its output, not from its input.
4. Predictor \(g_\phi\), a narrow ViT (width 384, depth 6 to 16 depending on the encoder) that receives the context tokens plus a learnable mask token carrying the positional embedding of each target patch, once per target block.

The loss averages the squared distance over the \(M\) blocks and their patches:

where \(B_i\) is the set of patch positions in block \(i\), \(\hat s_{y_j}\) the predictor's output at position \(j\), and \(s_{y_j}\) the target encoder's output there. Only \(\theta\) and \(\phi\) receive gradients. (The official code normalizes the targets with a parameter-free layer norm and uses a smooth L1 loss; the paper writes plain L2².)

The masks are the part people underestimate. Try the paper's settings and then shrink the targets or the context:

The paper's ablations, all on a linear probe trained with 1% of ImageNet labels (ViT-B/16 unless noted), measure how much the masking matters²:

The targets have to be large enough to be semantic — a block a quarter of an object's size cannot be guessed from its neighbouring texture — and the context spread enough to be informative.

One training step, line by line

The panel runs one step of the training loop on a toy image: 6 patches of 2 numbers, embeddings of size 3, a context of 3 patches and 2 targets. A ViT would mix the patches with attention; here each patch is encoded on its own and the predictor averages the context, so that every number fits on the screen:

The default case is the worked example. The context encoder maps patch 0, \([1, 0]\), to \([1, 0, 1]\); the target encoder maps patch 4, \([0, 2]\), to \([0, 2.2, -2]\), which layer norm turns into \([-0.039, 1.244, -1.205]\). The predictor's guess for that position is the context mean \([0.667, 0.667, 0]\) plus its learned position vector \([0, 1, -1]\): \([0.667, 1.667, -1]\). Over the six numbers of the two targets the loss is 0.6162, and one gradient step with learning rate 0.5 brings it to 0.2214 — while the target encoder's gradient is None.

Then pick the collapsed case. Every weight is zero, every embedding is zero, layer norm of a zero vector is zero, and so the loss is exactly 0 and every gradient is exactly 0. That is the whole collapse problem in one screen: the trivial solution is a perfect, stable minimum of the loss, and only the dynamics keep training away from it.

What it bought

I-JEPA's headline was efficiency without hand-crafted augmentations. Its ViT-H/14 trained on ImageNet-1K in under 72 hours on 16 A100 GPUs — under 1 200 GPU hours, more than ten times less than an MAE ViT-H/14². Linear-probe accuracy on ImageNet-1K, from the paper's Table 1:

Against pixel prediction the comparison is direct: the same ViT-L/16 recipe with pixel targets reaches 40.7% on the 1%-label probe after 800 epochs, against 66.9% after 500 with representation targets².

Why predicting representations helps — and when it does not

A theory result makes the intuition precise. Littwin et al. analysed deep linear versions of both objectives¹⁷: a JEPA is biased toward high-influence features — the ones with large regression coefficients, i.e. those that predict the target well — while the order in which MAE learns is set by the feature covariance, so high-variance details come first whether or not they are predictive. The bias "takes hold only when the encoder is deep". Two consequences, both visible in the lab:

• A JEPA is not automatically better. Small encoders, short schedules and tiny images remove the regime where its bias pays.
• The bias is the point at scale: pixel variance in natural images is dominated by texture and lighting; the semantics are lower-variance and more predictable.

Video: V-JEPA and V-JEPA 2

Video is where the "do not predict every pixel" argument is strongest, and where the JEPA line went next.

V-JEPA (2024)⁵ applies the recipe to clips of 16 frames: tokens are tubelets of 2 frames × 16 × 16 pixels, the masks are blocks repeated across all frames (about 90% of the tokens hidden), the loss is L1, and the predictor is a 12-block, 384-wide ViT. Trained on about 2 million videos, evaluated with the encoder frozen and an attentive probe, its ViT-H/16 reaches 82.0% on Kinetics-400 and 71.4% on Something-Something v2 — the benchmark where motion matters, and where image models trail by about 20 points (DINOv2 ViT-g 50.6%, I-JEPA 50.0%). Against pixel targets with the same model and budget: 73.7 vs 68.6 on Kinetics-400, 74.8 vs 73.3 on ImageNet⁵.

V-JEPA 2 (2025)⁶ scales it: more than a million hours of video plus a million images, a 1-billion-parameter ViT-g encoder, 3-D rotary positions, and a progressive-resolution schedule that cuts the training cost 8.4 times. The frozen encoder reaches 77.3% on Something-Something v2 and 39.7 recall@5 on Epic-Kitchens-100 action anticipation, against 27.6 for the previous best. Aligned with an 8-billion-parameter language model, it answers video questions at 84.0 on PerceptionTest.

V-JEPA 2-AC: the same energy, used to plan

The second half of the V-JEPA 2 paper is the bridge to the next class. Freeze the encoder. Train a new action-conditioned predictor — about 300 million parameters, block-causal attention — on less than 62 hours of unlabeled robot video from the Droid dataset: given the representation of the current frame, the robot's end-effector state and an action (a 7-D change in position, rotation and gripper), predict the representation of the next frame. That predictor is a world model, and the JEPA energy becomes a planning cost:

where \(\hat s_T\) is the predicted representation after the actions and \(s_{\text{goal}}\) is the encoder's representation of a goal image. A tiny example: the goal embedding is \([1, 0, 2]\) and three candidate actions lead to predicted embeddings \([0.5, 0, 1.5]\), \([1, 0.5, 2]\) and \([2, 0, 2]\). Their L1 energies are \(0.5 + 0 + 0.5 = 1.0\), \(0 + 0.5 + 0 = 0.5\) and \(1 + 0 + 0 = 1.0\): take the second. V-JEPA 2-AC searches with the cross-entropy method — sample 800 actions, keep the 10 with the lowest energy, refit the sampling distribution, repeat 10 times — executes the first action, observes, and plans again (receding-horizon control).

Deployed zero-shot on Franka arms in two labs it had never seen, it reached 65% at grasping a cup and 80% at pick-and-place with a cup, against 15% for Octo, a generalist robot policy fine-tuned on all of Droid. Planning took 16 seconds per action, against 4 minutes for Cosmos, a video-generation world model, run on the same GPU⁶.

V-JEPA 2.1 (2026) added a dense predictive loss so that patch-level features are also useful (depth estimation, navigation), reached 77.7% on Something-Something v2, and improved the cup-grasping rate from 60% to 80% with a longer planning horizon¹⁸. A separate study found that V-JEPA, tested with violation-of-expectation videos, distinguishes physically possible from impossible events — 98% zero-shot on IntPhys — where pixel-prediction models and multimodal LLMs were near chance¹⁹.

The regularized route: LeJEPA and SIGReg

The distillation recipe works, but it is a stack of heuristics: stop-gradient, EMA schedules, predictor depth, layer norm on the targets. LeJEPA (Balestriero and LeCun, 2025)¹³ asks what the embedding distribution should be, and proves that, for minimizing the risk of the predictors trained on top of it downstream, the answer is an isotropic Gaussian. It then enforces that distribution directly with SIGReg (Sketched Isotropic Gaussian Regularization): project the batch of embeddings on random unit directions \(a\), and for each 1-D projection measure how far its distribution is from \(\mathcal N(0, 1)\) with the Epps–Pulley statistic — the weighted squared distance between its empirical characteristic function and the Gaussian's, \(e^{-t^2/2}\):

The paper's code evaluates the integral with 17 points on \([-5, 5]\) and 256 directions per step. The total loss is \((1-\lambda)\,\mathcal L_{\text{pred}} + \lambda\,\text{SIGReg}\) with a single hyperparameter, \(\lambda = 0.05\) by default. No stop-gradient, no teacher–student, no schedulers; about 50 lines of code; 79% linear-probe accuracy on ImageNet-1K with a ViT-H/14. Where VICReg constrains two moments (variance and covariance), SIGReg constrains the whole distribution.

The same regularizer is what made the first JEPA world model trainable end-to-end from raw pixels with two loss terms: LeWorldModel (2026), 15 million parameters, one GPU, planning with the cross-entropy method up to 48 times faster than world models built on foundation-model features²⁰.

Beyond vision

The idea is not tied to pixels. LLM-JEPA (2025)²¹ adds a JEPA term to language-model training: a natural-language description and its code (a regex, an SQL query) are two views of the same thing, and the model is asked to predict the embedding of one from the other, alongside the usual next-token loss — with gains reported across Llama 3, Gemma 2, OpenELM and OLMo models. VL-JEPA (2025)²² predicts the embedding of the answer text instead of its tokens, decoding words only when needed, with half the trainable parameters of a comparable token-space model.

MAE, CLIP and JEPA side by side

Toward world models

LeCun's position paper was not about image features. It describes an agent made of six modules — perception, a world model, a cost, an actor, a short-term memory and a configurator — that plans by running the world model forward and choosing the actions with the lowest cost, which is model-predictive control with a learned model and a learned cost¹. The JEPA was proposed as the world model, and H-JEPA stacks them: a low level predicting the next fraction of a second in detail, a higher level predicting further ahead in more abstract terms.

V-JEPA 2-AC is a working instance of that loop, and its weak point is visible in the equation above: the cost is a plain L1 distance to a goal image's representation. Two states that look alike can be many actions apart. The next class takes world models on their own terms; the one after it, TDWM (2026), answers exactly this weakness from the side of the representation: its loss makes the similarity of two frames fall with the number of steps between them, so that distance in the latent measures time rather than looks. Its closest neighbour, Temporal-Distance JEPA²³, keeps the representation and learns a separate cost that counts the steps.

Lab: a JEPA, built and measured

The Colab notebook trains a small I-JEPA on CIFAR-10 and puts every claim of this page to a measurement. Its training runs took about 18 minutes in total on the Apple M2 GPU it was checked on; a free Colab T4 is in the same range. Nine sections, each ending in a checkpoint with check values:

1. the multi-block mask on an 8 × 8 patch grid;
2. the three networks — context encoder, target encoder, predictor — and their sizes;
3. one training step by hand, and proof that the target encoder gets no gradient;
4. how to measure a representation without labels (standard deviation, effective rank) and with them (linear probe), on raw pixels and on an untrained encoder;
5. I-JEPA, trained;
6. collapse, produced on purpose: remove the stop-gradient, then remove the EMA;
7. the regularized route: no EMA, no stop-gradient, a variance–covariance term;
8. pixels instead of representations: a small MAE at the same budget;
9. an open task.

What it measured, on CIFAR-10 with a 4-block ViT of 0.8 M parameters (linear probe on 10 000 training images, accuracy on the 10 000 test images; spread is the standard deviation of the normalized embeddings times \(\sqrt d\)):

Three readings. The two collapsed runs have by far the lowest losses of the notebook and the worst representations — and only the spread catches both, since the effective rank is blind to a representation that simply shrank. The variance–covariance terms prevent collapse completely (the highest rank of all) and still do not give good features: preventing collapse is necessary, not sufficient. And at this scale the MAE beats the I-JEPA by four points: the regime where predicting representations pays — deep encoders, large images, long training — is exactly what a 2-hour lab cannot reach.

Open the lab in Colab — save a copy to your Drive (File → Save a copy in Drive) to run and edit it.

Key takeaways

1. A JEPA predicts the representation of the hidden part from the representation of the visible part. The energy is the prediction error in representation space, so details that cannot be predicted need not be modelled.
2. Any joint embedding can collapse: a constant encoder makes every target predictable and the loss zero. A falling loss is no evidence of learning; measure the spread and the effective rank.
3. Four ways out: negatives (SimCLR, MoCo, CLIP), asymmetry with stop-gradient, predictor and EMA target (BYOL, DINO, I-JEPA), variance–covariance terms (VICReg), distribution matching (LeJEPA's SIGReg).
4. I-JEPA: a context block, four large target blocks, targets cut from the output of an EMA encoder that saw the whole image. The masking choices are worth tens of points.
5. Its gain is frozen features per unit of compute without augmentations, not a uniform win: augmentation-based methods and MAE still lead on some tasks, and at small scale pixel reconstruction can be better.
6. In video the argument is strongest: V-JEPA beats pixel prediction at the same budget, video features lead image features by about 20 points where motion matters, and V-JEPA 2-AC turns the energy into a planning cost for a real robot.
7. The JEPA was proposed as a world model. The next class starts there.