VL-JEPA: Joint Embedding Predictive Architecture for Vision-language

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Novel JEPA architecture for VLMs; continuous embedding prediction vs. autoregressive tokens

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VL-JEPA replaces autoregressive token generation in VLMs with latent-space embedding prediction using a Joint Embedding Predictive Architecture (JEPA). Under controlled comparisons, it achieves stronger performance than token-generative VLMs with 50% fewer trainable parameters, and natively supports selective decoding that reduces decoding operations by ~2.85× for streaming video applications.

Problem

Standard generative VLMs are training-inefficient because they must model both task-relevant semantics and task-irrelevant surface linguistic variability (word choice, style, paraphrasing) in the raw one-hot token space. Additionally, autoregressive decoding is incompatible with low-latency streaming inference, since the full token sequence must be decoded before semantics are accessible.

Method

VL-JEPA uses four components: (1) an X-Encoder (frozen V-JEPA 2 ViT-L, 304M params) mapping visual input \(X_V \mapsto S_V\); (2) a Predictor (8 Llama-3.2-1B Transformer layers, 490M params) computing \((S_V, X_Q) \mapsto \hat{S}_Y\); (3) a Y-Encoder (EmbeddingGemma-300M) computing \(Y \mapsto S_Y\); (4) a lightweight Y-Decoder invoked only at inference when text output is required, computing \(\hat{S}_Y \mapsto \hat{Y}\).

The training objective operates in a shared 1,536-dimensional embedding space via bi-directional InfoNCE loss — decomposable into an alignment term (minimizing distance between \(\hat{S}_Y\) and \(S_Y\)) and a uniformity regularization term (anti-collapse). This replaces the token-space cross-entropy \(\mathcal{L}_\text{VLM} = D(\hat{Y}, Y)\) with \(\mathcal{L}_\text{VL-JEPA} = D(\hat{S}_Y, S_Y)\), collapsing multiple valid paraphrases of the same answer to nearby points rather than orthogonal token sequences.

Training is two-stage: (1) large-scale query-free pretraining on caption data (Datacomp, YFCC-100M, ACTION-100M; 3.6B samples seen, 4 weeks on 192×H200); (2) query-conditioned SFT on ~30M VQA/captioning/classification samples from the PLM data mixture.

Selective decoding for streaming: the continuous \(\hat{S}_Y\) stream is smoothed (average pooling over local windows) and the Y-Decoder is triggered only when variance exceeds a threshold, enabling always-on semantic monitoring without per-frame autoregressive decoding.

Key Contributions

• First non-generative VLM for general-domain vision-language tasks built on JEPA, operating in continuous embedding space rather than token space.
• Controlled ablation demonstrating that embedding-space supervision outperforms token-space supervision under matched training conditions (same encoder, data, iterations, batch size) with half the trainable parameters (0.5B predictor vs. 1B LLM).
• Native selective decoding that reduces decoding operations by ~2.85× on streaming video while maintaining CIDEr parity.
• Unified architecture handling classification, text-to-video retrieval, discriminative VQA, and generative captioning without architectural modification.
• State-of-the-art on WORLD PREDICTION-WM (65.7%), outperforming GPT-4o, Claude-3.5-Sonnet, and Gemini-2.0 on inverse dynamics reasoning.

Results

Zero-shot classification (8 video datasets, avg. Top-1): - VL-JEPA\(_\text{BASE}\): 52.5 vs. PE-Core-G: 44.7 (best prior generalist, trained on 86B samples vs. 3.6B for VL-JEPA)

Zero-shot text-to-video retrieval (8 datasets, avg. Recall@1): - VL-JEPA\(_\text{BASE}\): 63.7 vs. PE-Core-G: 58.1

VQA (VL-JEPA\(_\text{SFT}\), 1.6B params): - GQA: 61.5 (vs. InstructBLIP Vicuna-13B: 49.5; below LLaVA-1.5 Vicuna-7B: 62.0) - TallyQA: 69.9 (vs. InstructBLIP Vicuna-13B: 68.0; below PaliGemma 3B: 76.8) - POPE: 85.7 (vs. InstructBLIP Vicuna-13B: 79.0; comparable to LLaVA-1.5-7B: 85.9) - POPEv2: 86.3 (vs. Qwen2-VL-7B: 87.0; below Qwen2-VL-2B: 91.3)

WORLD PREDICTION-WM (inverse dynamics): - VL-JEPA\(_\text{SFT}\): 65.7% (new SoTA; GPT-4o: 52.7, Claude-3.5-Sonnet: 53.3, Gemini-2.0: 55.6)

Selective decoding: - ~2.85× reduction in decoding calls with similar CIDEr performance vs. uniform decoding.

Controlled comparison (embedding vs. token prediction): - VL-JEPA (0.5B predictor) consistently outperforms VLM baseline (1B LLM) on captioning CIDEr and classification Top-5 across all checkpoints from 500K to 15M samples.

Limitations

• VL-JEPA\(_\text{BASE}\) is weaker on appearance-centric benchmarks (Kinetics-400, COIN task recognition) relative to motion-centric ones, attributed to significantly fewer training samples (3.6B vs. 86B for PE-Core-G).
• Generative decoding quality depends on the lightweight Y-Decoder; performance on open-ended generation benchmarks relative to larger generative VLMs is not fully characterized.
• InfoNCE contrastive loss requires large batch sizes; more advanced non-contrastive regularization (VICReg, SIGReg) is deferred to future work.
• VQA results do not consistently surpass all baselines — POPEv2 scores fall below several Qwen2-VL variants despite them being larger models; GQA is below LLaVA-1.5-7B.
• The Y-Decoder is a separate component whose quality and training procedure are underspecified in the text excerpt.

Relevance to Vision-Language Models

VL-JEPA directly challenges the prevailing paradigm that VLMs must autoregressively generate tokens, showing that latent-space prediction is both more parameter-efficient and more semantically precise for discriminative and retrieval tasks. The controlled comparison is methodologically significant: it isolates the training objective (embedding vs. token prediction) from all other variables, providing strong evidence that the embedding space simplifies the learning target by collapsing paraphrase-diverse outputs. For researchers tracking VLMs, the selective decoding mechanism is practically important for streaming and embodied applications where autoregressive decoding latency is prohibitive. The JEPA framing also reconnects VLMs to the broader self-supervised representation learning literature (I-JEPA, V-JEPA 2), suggesting a path toward unified architectures that handle perception and generation in a shared abstract space.