V-JEPA understood video beautifully but its feature maps were a mess. The fix is one extra loss term — and it turns a world model into a dense one.
↓ SCROLL
01 — The symptom
V-JEPA 2 understands what is happening in a video better than almost anything. Ask it where things are, though, and the feature maps come back as noise.
V-JEPA 2
PCA of patch features, top three components → RGB
Fragmented. Neighbouring patches of the same object land in unrelated places, and the map flickers with artefacts.
V-JEPA 2.1
same visualisation, same scene
Coherent. Similar parts map to similar components — the dog's head, the car's wheels — and the regions hold together.
and it shows up in the numbers
A single linear layer on top of the frozen encoder. This is the standard way to ask what a representation already contains.
Semantic segmentation · ADE20K
mIoU, higher is better
Depth estimation · NYUv2
RMSE, lower is better
Why this matters beyond pretty pictures. Dense features are what a robot needs to reach for a cup, what a planner needs to judge distance, and what any segmentation or tracking head reads. A world model that cannot localise is a world model you cannot act with.
02 — The diagnosis
The original loss is applied only to the masked tokens. So the predictor never has to keep the visible ones grounded — and the encoder is free to use them as scratch space for global summaries instead.
Note the index set. The predictor emits an output for every token it is given — context tokens included — but only the masked ones ever appear in the sum. The rest are free.
unsupervised context
With nothing holding them in place, visible tokens drift into acting like register tokens — global aggregators that carry summary information rather than local content.
with the context loss
Now every token is asked to match the target encoder's output at its own position. It has to stay about what is actually there.
The one-sentence hypothesis
High-quality dense features do not emerge when the prediction loss is applied only to masked regions — so extend it to the entire input, masked and visible alike.
03 — Ingredient one
Apply the same objective to the visible tokens. Do it naively and the model finds a shortcut — so the weight on each context token depends on how far it sits from the nearest hidden region.
the context loss
Identical to the original objective except for the index set — C, the context tokens — and the per-patch weight λᵢ. The full objective is simply their sum.
the weighting
dmin is the distance, in blocks, from a context token to the nearest masked token. Patches on the boundary of a hidden region get the most weight; patches far away get little. The effect is to enforce continuity across the seam between what was hidden and what was not.
λ itself is warmed up over epochs 50–100, and set to 0.5 for video and 0.7 for images.
the trade-off, measured — Table 2
Up is better segmentation. Right is better action recognition. Click a point.
what λᵢ looks like on the grid
Distance-weighted: patches next to a hidden region count for more.
selected
λ = 0.5 distance-weighted, warmed up
The chosen configuration. Weighting by the inverse square root of the distance to the nearest masked block enforces local continuity, and recovers most of the action-recognition performance the naive version threw away.
ADE20K mIoU
33.8
+11.6 vs V-JEPA 2
SSv2 accuracy
62.5
-10.3 vs V-JEPA 2
Read the scatter carefully. The dashed path is what happens if you just turn λ up: segmentation peaks around 0.2 and then falls, while action recognition drops from 72.8 all the way to 51.1. Every later ingredient in this paper exists to buy that accuracy back.
04 — Ingredient two
Do not only supervise the encoder's last layer. Tap four depths, fuse them, predict from the fusion — and apply both losses at every level.
what it buys
Adding the context loss cost 10 points of action recognition. Deep supervision gets almost all of it back, and improves segmentation at the same time.
why it works
Supervising only the last layer leaves the early and middle blocks to be shaped indirectly. Tapping four depths gives every part of the network a direct training signal — and lets local information flow forward to the final layers instead of being discarded on the way.
a practical side effect
Because the last layer now carries local detail too, downstream dense tasks no longer need to reach into intermediate layers to find it. You can just use the output.
05 — Ingredients three and four
V-JEPA 2 fed images through a 3D convolution by copying them into sixteen identical frames — expensive, and a lie about what an image is. Fix that, widen the data, and grow the model.
V-JEPA 2 · one 3D convolution for everything
An image was duplicated sixteen times and treated as a static video.
V-JEPA 2.1 · one tokenizer per modality
Each input is processed in its native form, with a learnable token saying which pathway it came from.
ADE20K improves 40.8 → 41.4 mIoU, recognition holds steady, and images stop costing sixteen times what they should.
VisionMix 163M · rebalancing the diet
ImageNet's one million images are replaced by the 142-million-image LVD-142M set. With static appearance now well covered, the video sampling shifts hard toward motion-rich sources.
SSv2
Ego video
168K
V-JEPA 2 · 0.056
V-JEPA 2.1 · 0.170
Kinetics
Exo video
733K
V-JEPA 2 · 0.188
V-JEPA 2.1 · 0.010
HowTo100M
Exo video
1.1M
V-JEPA 2 · 0.318
V-JEPA 2.1 · 0.100
YT-1B
Exo video
19M
V-JEPA 2 · 0.188
V-JEPA 2.1 · 0.720
ImageNet
Images
1M
V-JEPA 2 · 0.250
V-JEPA 2.1 · 0.000
LVD-142M
Curated images
142M
V-JEPA 2 · 0.000
V-JEPA 2.1 · new
Images and videos are not mixed inside a batch. Separate workers handle each modality and their gradients are aggregated — 2,304 images and 128 clips per global step.
06 — The algorithm
Two of these lines are new since V-JEPA 2. They are lines 11 and 12.
V-JEPA 2.1 · one step
line 1
Images and videos are drawn on separate workers and their gradients aggregated — 2,304 images and 128 clips per global batch.
what is active
Pretraining settings
Appendix A · identical for ViT-L, g and G
| Steps | 135,000 + 12,000 | primary phase, then cool-down |
| Learning rate | 1e-4 → 5.25e-4 → 1e-6 | 12k warm-up, constant, then decayed in cool-down |
| Video batch | 128 clips | 16 frames at 4 fps, 256×256 |
| Image batch | 2,304 | 256×256; separate workers per modality |
| Cool-down video | 64 frames · 384² | 12k extra steps |
| Cool-down image | 512² | |
| Patch size | 16 | tubelet size 2 for video |
| Predictor | 24 blocks · dim 384 | V-JEPA 2 used 12 |
| Deep supervision | 4 levels | equally spaced encoder blocks |
| λ (context loss) | 0.5 video · 0.7 image | distance-weighted, warmed up |
| EMA coefficient | 0.99925 | fixed |
| Weight decay | 0.04 | fixed |
07 — The ablation
Table 1, one ingredient at a time. Press play, or step through it — the shape of this chart is the whole argument of the paper.
starting point
starting point
starting point
starting point
step 1 of 7 · the starting point
V-JEPA 2
Strong global video understanding. The loss is applied only to the masked tokens.
what happens to the numbers
Dense features are noisy and fragmented — 22.2 mIoU on ADE20K is close to unusable for segmentation.
The context loss alone is a bad trade: dense tasks improve a lot, global tasks collapse. If the paper stopped there it would not be publishable.
Four supervised depths recover almost all the lost accuracy while dense keeps climbing. This is the step that makes the idea viable.
More data, a proper tokenizer, ten times the parameters, and a high-resolution cool-down. Every one of them helps, and none of them needs the recipe rethought.
Worth stating plainly: the ablation runs on a ViT-L and the two scaling rows change the model and the schedule, not just a hyperparameter. So the last two jumps are not like-for-like with the first four — the paper presents them in the same table, and it is worth reading them as a separate claim.
08 — Results
Everything below uses a frozen encoder. The claim is about the representation, not about what a fine-tuning run can recover.
Dense visual tasks
Depth (RMSE, lower better), segmentation (mIoU), video object segmentation (J&F). Table 8.
| Param | NYUv2 ↓ | KITTI ↓ | ADE20K | Citysc. | VOC | DAVIS | YT-VOS | |
|---|---|---|---|---|---|---|---|---|
| Web-DINO | 7B/14 | 0.466 | 3.158 | 42.7 | 68.3 | 76.1 | 57.2 | 43.9 |
| DINOv3 | 7B/16 | 0.309 | 2.346 | 55.9 | 81.1 | 86.6 | 71.1 | 74.1 |
| PEspatial | 2B/14 | 0.362 | 3.082 | 49.3 | 73.2 | 82.7 | 68.4 | 68.5 |
| AM-RADIOv2.5 | 1B/14 | 0.340 | 2.918 | 53.0 | 78.4 | 85.4 | 66.5 | 70.1 |
| DINOv2 | 1B/14 | 0.372 | 2.624 | 49.5 | 75.6 | 83.1 | 63.9 | 65.6 |
| DINOv3 ViT-H+ | 0.8B/16 | 0.352 | 2.635 | 54.8 | 79.5 | 85.8 | 71.1 | 74.0 |
| V-JEPA 2 ViT-gprevious | 1B/16 | 0.642 | 4.650 | 24.4 | 45.9 | 63.9 | 52.5 | 53.7 |
| V-JEPA 2.1 ViT-g | 1B/16 | 0.350 | 2.601 | 47.8 | 71.8 | 84.7 | 68.1 | 72.3 |
| V-JEPA 2.1 ViT-G | 2B/16 | 0.307 | 2.461 | 47.9 | 73.5 | 85.0 | 69.0 | 72.7 |
the row worth pausing on
V-JEPA 2 scored 0.642 RMSE on NYUv2 depth and 24.4 mIoU on ADE20K — near the bottom of the table. V-JEPA 2.1 at the same 1B size scores 0.350 and 47.8. The 2B model then beats DINOv3 ViT-7B on depth while being three and a half times smaller.
Global recognition
Top-1 accuracy. Motion-centric tasks on the left, appearance-centric on the right. Table 9.
| Param | SSv2 | Diving-48 | K400 | IN1K | |
|---|---|---|---|---|---|
| DINOv3 | 7B | 70.1 | – | 87.8 | 88.4 |
| DINOv2 | 1.1B | 50.7 | 82.5 | 83.6 | 86.1 |
| PEcore G | 1.9B | 55.4 | 76.9 | 88.5 | 87.6 |
| SigLIP2 | 1.2B | 49.9 | 75.3 | 87.3 | 88.0 |
| InternVideo2 | 1B | 69.7 | 86.4 | 89.4 | 85.8 |
| V-JEPA ViT-H | 600M | 74.3 | 87.9 | 84.5 | 80.0 |
| V-JEPA 2 ViT-gprevious | 1B | 77.3 | 90.2 | 87.3 | 85.1 |
| V-JEPA 2.1 ViT-g | 1B | 76.9 | 89.0 | 87.0 | 84.8 |
| V-JEPA 2.1 ViT-G | 2B | 77.7 | 89.2 | 87.7 | 85.5 |
Predicting where an interaction will happen, with what object, doing what, and when — all four have to be right. 7.71 mAP is a 35% relative gain over the previous best, which used task-specific training components.
real robot · zero-shot grasping
Closed-loop manipulation on a real Franka arm, zero-shot in a new environment. Model-predictive control with the cross-entropy method on one A100.
Grasp success on a Franka arm in an environment it has never seen: 60% → 80%. Reach and pick-and-place are unchanged. The authors note the remaining failures are gripper timing, not spatial misunderstanding.
navigation planning
Open-loop navigation planning. A latent world model trained on top of frozen features, following the NWM setup.
| Time | Recon | Tartan Drive | Scand | Sacson | Average | |
|---|---|---|---|---|---|---|
| NWM | 103.2 s | 1.146 | 5.831 | 1.113 | 4.037 | 3.032 |
| V-JEPA 2.1 ViT-g | 10.6 s | 1.146 | 5.758 | 1.094 | 3.904 | 2.975 |
| V-JEPA 2.1 ViT-G | 10.6 s | 1.179 | 5.687 | 1.038 | 4.054 | 2.990 |
Absolute trajectory error, lower is better. The headline is the first column: planning in this representation needs 8 denoising steps where the previous approach needed at least 128 — a 10× speed-up at equal or better accuracy.
Distillation into smaller models
The 2B teacher compressed into 300M and 80M students. Table 11.
| Param | SSv2 | K400 | IN1K | ADE20K | NYU ↓ | KITTI ↓ | DAVIS | |
|---|---|---|---|---|---|---|---|---|
| ViT-G | 2B | 77.7 | 87.7 | 85.5 | 47.9 | 0.307 | 2.461 | 69.0 |
| ViT-L | 300M | 74.2 | 84.3 | 81.8 | 42.0 | 0.381 | 2.914 | 64.5 |
| ViT-L (distilled) | 300M | 76.5 | 86.9 | 84.9 | 46.7 | 0.336 | 2.490 | 68.7 |
| ViT-B (distilled) | 80M | 74.5 | 85.1 | 84.0 | 42.0 | 0.408 | 2.850 | 67.0 |
The distilled ViT-L at 300M recovers most of the 2B teacher — 76.5 against 77.7 on SSv2, 46.7 against 47.9 mIoU — and comfortably beats the same architecture trained from scratch. Distillation drops deep self-supervision and uses a frozen teacher with a 12-block predictor.
09 — What it costs
Dense features are bought with something. Comparing the two models of the same size makes the price visible.
like for like · both 1B parameters
V-JEPA 2 ViT-g against V-JEPA 2.1 ViT-g, on global benchmarks.
Compare the two 1B models. On every global benchmark the new recipe is a fraction behind V-JEPA 2 — and behind by more than a fraction on EPIC-KITCHENS. The state-of-the-art numbers all come from the 2B model. Dense features are not free.
TemporalBench and TOMATO both drop. Both test understanding of motion events in short videos — exactly the thing V-JEPA was already good at, and the thing the context loss pulls capacity away from.
24 blocks against V-JEPA 2's 12, plus a fusion MLP and four prediction heads. The pretraining is more expensive per step than the method it replaces.
0.5 for video, 0.7 for images, with a warm-up schedule found empirically. The paper is candid that naive application 'leads to poor performance' and that the weighting exists to stop the model copying context features.
the claim, stated exactly
Supervising the visible tokens, with a weight that falls off with distance from the mask, turns V-JEPA's fragmented feature maps into coherent ones — 22.2 → 47.9 mIoU, 0.682 → 0.307 RMSE, and a 20-point jump in real-robot grasp success. Deep self-supervision is what makes that affordable. At matched model size it is still a small step backwards on global recognition; the state-of-the-art results come from scaling to 2B parameters. Both halves are in the tables.