Skip to content

LeVLJEPA: End-to-End Vision-Language Pretraining Without Negatives

🕒 Published (v1): 2026-07-01 11:15 UTC · Source: Arxiv · link

Why this paper was selected

Non-contrastive VL pretraining without negatives — architectural shift away from CLIP-style objectives

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LeVLJEPA is the first fully non-contrastive end-to-end vision-language pretraining method, replacing the contrastive InfoNCE/SigLIP objective with cross-modal prediction under stop-gradient targets and per-modality SIGReg distributional regularization. It trains stably with no negatives, temperature, momentum encoder, or teacher-student schedule, and produces patch-token features that outperform CLIP and SigLIP when used as a frozen visual backbone in VLMs and for semantic segmentation.

Problem

Vision-language pretraining (CLIP, SigLIP, VL-JEPA) remains dominated by contrastive objectives that optimize a single pooled image-text alignment signal, while vision-only SSL has moved to non-contrastive methods (DINO, I-JEPA, LeJEPA). Contrastive objectives have two compounding drawbacks: (1) performance scales with batch size via in-batch negatives, requiring large-batch infrastructure; (2) they supervise per-patch token features only as a byproduct of pooled-embedding alignment, whereas downstream VLMs and dense prediction systems consume the full grid of patch tokens. No non-contrastive method existed for joint vision-language pretraining.

Method

LeVLJEPA extends the LeJEPA framework to the cross-modal setting. The key design choices:

Why direct symmetric MSE fails. Replacing the contrastive term with \(L_{\text{direct}} = \frac{1}{B}\sum_i \|z_i^v - z_i^t\|_2^2 + \lambda_v\,\text{SIGReg}(Z^v) + \lambda_t\,\text{SIGReg}(Z^t)\) collapses: both encoders meet in the shared low-dimensional subspace defined by the coarse image-caption overlap, because gradients flow into both encoders symmetrically from the same target. SIGReg alone improves effective rank but cannot recover the cross-modal geometry.

Asymmetric cross-modal prediction. Two modality-specific MLP predictors \(h^v: \mathbb{R}^d\to\mathbb{R}^d\) and \(h^t: \mathbb{R}^d\to\mathbb{R}^d\) predict the opposite modality's embedding with stop-gradients on the target: $\(L_{\text{cross}} = \frac{1}{B}\sum_i \left[\|h^v(z_i^v) - \text{sg}(z_i^t)\|_2^2 + \|h^t(z_i^t) - \text{sg}(z_i^v)\|_2^2\right]\)$ Stop-gradient decouples each encoder's gradient path so SIGReg can shape each modality's marginal distribution without interference from the symmetric cross-modal term.

SIGReg (Sketched Isotropic Gaussian Regularization). Rather than matching densities directly in \(\mathbb{R}^d\), SIGReg projects embeddings onto random 1D directions and applies an Epps–Pulley characteristic-function normality test to each projection, enforcing an isotropic Gaussian marginal in \(O(Bd)\) time and memory.

Full objective: $\(L_{\text{LeVLJEPA}} = (1-\lambda_v-\lambda_t)\,L_{\text{cross}} + \lambda_v\,\text{SIGReg}(Z^v) + \lambda_t\,\text{SIGReg}(Z^t)\)$

Architecture. ViT-B/16 image encoder + GPT-2 text encoder; each backbone output is projected through a one-layer MLP (hidden dim 2048, GELU, BatchNorm, Dropout) before SIGReg. Cross-modal predictors \(h^v, h^t\) operate on top of these projections. BatchNorm in the projection is necessary because final LayerNorm in both ViT and GPT-2 would prevent SIGReg from being optimized.

Batch size invariance. Because \(L_{\text{cross}}\) is computed per matched pair and SIGReg acts on the empirical marginal, LeVLJEPA's performance is insensitive to batch size \(B \in \{1024, 2048, 4096\}\), unlike CLIP which improves monotonically with \(B\).

Training. Two-stage protocol: hyperparameters selected on CC12M (12M samples), then a single fixed configuration evaluated on Datacomp-L (92M samples, 819M samples seen, ViT-B/16 backbone shared across all methods).

Key Contributions

  • First fully non-contrastive end-to-end vision-language pretraining method, requiring no negatives, temperature, momentum encoder, or teacher-student schedule.
  • Identification and diagnosis of symmetric cross-modal MSE collapse, and a concrete fix via asymmetric stop-gradient predictors.
  • Demonstration that batch-size invariance (unlike CLIP/SigLIP) can be achieved in the vision-language setting.
  • Evidence that non-contrastive pretraining produces stronger dense per-token semantic features than contrastive objectives, at the cost of weaker zero-shot alignment.

Results

All comparisons use ViT-B/16, identical data (Datacomp-L, 92M downloadable samples), identical samples-seen budget (819M). Baselines are InfoNCE (CLIP-style) and SigLIP with standard OpenCLIP hyperparameters.

Zero-shot classification (Datacomp-L): LeVLJEPA trails contrastive baselines — ImageNet: 42.45% vs. 47.32% (InfoNCE) / 50.78% (SigLIP); Pets: 59.63% vs. 68.98% / 77.27%. Gap is expected since zero-shot scores matched vs. unmatched pairs, the objective contrastive methods directly optimize.

Linear probing on frozen CLS token (Datacomp-L): On par across all benchmarks — ImageNet: 65.42% vs. 65.75% / 66.34%; Aircraft: 46.38% vs. 44.10% / 47.46%. Parity holds at CC12M scale (31.84% vs. 31.91% / 31.83%).

Background robustness (ImageNet-9, Datacomp-L): LeVLJEPA is best on all splits — Mixed-Rand: 79.75% vs. 77.31% (InfoNCE) / 78.35% (SigLIP); smallest accuracy drop under both Mixed-Same and Mixed-Rand shifts.

Dense features (frozen backbone in VLM, Datacomp-L): LeVLJEPA is the strongest encoder across GQA, VQAv2, and POPE under two distinct language-model families (specific numbers not present in the provided text excerpt).

Semantic segmentation (linear head on frozen patch tokens, ADE20K + COCO-Stuff): LeVLJEPA outperforms both contrastive baselines (specific mIoU numbers not present in the provided text excerpt).

Limitations

  • Trails contrastive baselines on zero-shot transfer at larger training scale (Datacomp-L); the gap narrows at smaller scale (CC12M) and smaller batch sizes.
  • Absolute numbers on Datacomp-L are not cross-paper comparable due to link rot reducing the available pool from ~140M to ~92M pairs; the gap to published CLIP/SigLIP numbers reflects data availability, not method quality.
  • Architecture relies on BatchNorm in the projection head to make SIGReg effective; this adds an implicit coupling to batch statistics not present in pure CLIP-style training.
  • Specific VQA and segmentation numbers are not reported in the provided excerpt, so the claimed margins on dense tasks cannot be verified from this text.
  • Evaluated only on ViT-B/16; generalization to larger backbones is not established.

Relevance to Vision-Language Models

LeVLJEPA directly challenges the assumption that contrastive pretraining is necessary for vision-language encoders, and proposes a concrete alternative that is architecturally simpler and infrastructure-friendlier (batch-size invariant). The finding that dense per-token features are stronger under non-contrastive pretraining is highly relevant to the dominant VLM paradigm (LLaVA-style), where a frozen visual encoder feeds patch tokens into a language model: the quality of those patch tokens, not the pooled CLS alignment, is what limits downstream VQA and captioning accuracy. This positions LeVLJEPA as a drop-in encoder replacement that could improve VLMs without changing the bridge/LLM architecture. More broadly, it closes the methodological gap between vision-only SSL (which has largely abandoned contrastive objectives) and vision-language pretraining, opening the door for JEPA-style predictive objectives across modalities.