Skip to content

Predictive Regularization Against Visual Representation Degradation in Multimodal Large Language Models

🕒 Published (v1): 2026-03-21 13:10 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; predictive regularization prevents visual representation degradation during LLM training

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MLLMs trained purely on next-token prediction progressively degrade their internal visual representations in intermediate LLM layers — a phenomenon the authors term visual representation degradation. They diagnose it at both the global (linear-probe classification) and patch (semantic boundary blurring) level, then propose PRe (Predictive Regularization): a lightweight 2-layer MLP that forces degraded intermediate features to predict the original visual anchor features via cosine similarity loss. Applying PRe yields consistent gains across multiple MLLM architectures and benchmarks without architectural changes.

Problem

Standard MLLM training optimizes exclusively for language generation (\(\mathcal{L}_{\text{LM}}\)), leaving visual representations without any direct visual objective. As a result, intermediate LLM layers transform patch features toward a "workspace" suited for abstract linguistic description — increasing geometric complexity and statistical independence of features — at the cost of losing fine-grained, visually-discriminative structure. Prior interpretability work studied how visual features serve language tasks, but the cost to the intrinsic visual quality of those features had not been systematically analyzed.

Method

Diagnostic analysis. For each LLM layer \(l\), the global aggregated feature \(\bar{h}^l = \frac{1}{N_p}\sum_i h^l_{v,i}\) is linearly probed on CUB-200 and ImageNet-100. Patch-level structure is quantified via Intra-Object Cohesion (mean cosine similarity within an object) and Inter-Object Coupling (mean similarity across objects); the Semantic Contrast ratio (Cohesion/Coupling) measures boundary sharpness. Both metrics consistently deteriorate in intermediate layers across LLaVA-style models.

PRe method. The initial projected visual tokens \(H^0_v\) serve as a stop-gradient anchor. A 2-layer MLP \(f_\text{pred}\) maps the degraded intermediate visual hidden states \(H^l_v\) (from the middle LLM layer) to predict the anchor:

\[\mathcal{L}_{\text{PRe}} = -\frac{1}{N_p} \sum_{i=1}^{N_p} \mathcal{D}\!\left(f_{\text{pred}}(h^l_{v,i}),\; \text{stopgrad}(h^0_{v,i})\right)\]

where \(\mathcal{D}\) is cosine similarity. The total loss is \(\mathcal{L}_\text{total} = \mathcal{L}_\text{LM} + \lambda\mathcal{L}_\text{PRe}\) with \(\lambda{=}0.5\). The degraded layer is fixed to the LLM midpoint (layer 16 for Vicuna-7B, layer 14 for Qwen-2.5-7B).

Key Contributions

  • First systematic multi-level diagnosis of visual representation degradation in MLLMs, linking macroscopic classification drops to microscopic patch semantic boundary blurring.
  • Causal account: degradation is a deliberate visual sacrifice shown by (a) statistical structure analysis (PCA effective dimension peaks while feature correlation drops at degraded layers) and (b) an inverse correlation between GQA accuracy and linear-probe accuracy tracked throughout pre-training.
  • PRe: a broadly applicable, architecture-agnostic regularizer that preserves visual fidelity via self-prediction from intermediate to initial visual features.

Results

All gains are relative to LLaVA-trained baselines under identical data (LLaVA-558K + LLaVA-665K) with Vicuna-7B-v1.5 or Qwen-2.5-7B-Instruct:

  • Vicuna + CLIP-ViT-L/14@336 (frozen encoder): GQA +0.7, MMMU +0.4, AI2D +1.7, MMStar +4.3, SQA +0.7, TextVQA +1.1, OCRbench +11.
  • Vicuna + SigLIP2-SO400M/14@384 (frozen encoder): GQA +1.8, AI2D +0.5, SQA +1.6, TextVQA +1.3, OCRbench +5.
  • Vicuna + SigLIP2 (trainable encoder): GQA +0.2, MMMU +0.3 (smaller gains with unfrozen encoder, as expected).
  • At the representation level, PRe restores linear-probe classification accuracy and Semantic Contrast ratio across all intermediate layers, not just the trained layer.

Limitations

  • Results are restricted to 7B-scale models; scalability to larger LLMs is not demonstrated.
  • The degraded layer is fixed heuristically to the LLM midpoint; sensitivity across deeper or shallower architectures is uncharacterized in available text.
  • The stop-gradient anchor is the projector output \(H^0_v\), which still captures visual encoder artifacts rather than raw pixel fidelity; true image-level grounding is not recovered.
  • The paper explicitly disclaims pursuit of SOTA, so absolute benchmark numbers may not fully reflect the method's ceiling.
  • \(\lambda\) tuning and its interaction with encoder freezing are ablated but may require per-dataset adjustment.

Relevance to Vision-Language Models

This work directly targets the core tension in the dominant MLLM training paradigm — optimizing purely for next-token prediction implicitly destroys visual structure that the model was initialized with — and provides both a rigorous diagnosis and a cheap fix. The finding that intermediate-layer visual features are representationally degraded challenges the common assumption that a strong frozen vision encoder is sufficient, and motivates auxiliary visual objectives in MLLM training. PRe's stop-gradient self-prediction design is plug-and-play over existing LLaVA-style pipelines, making it immediately applicable to the broader VLM ecosystem without new data or architectural overhaul. The causal link between visual fidelity and language task performance also has implications for understanding multimodal hallucination, visual grounding, and fine-grained VQA failures.