Skip to content

Elevating Visual Perception in Multimodal LLMs with Visual Embedding Distillation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VisPer-LM proposes distilling knowledge from expert vision encoders (depth, segmentation, generation) directly into the LLM's intermediate hidden representations during MLLM pre-training, rather than feeding their features as additional inputs. This vision-centric coupled objective—predictive embedding loss plus next-token prediction—improves spatial and depth perception without any inference-time overhead. On CV-Bench, it outperforms single-encoder and multi-encoder baselines with up to 8.7% improvement on depth tasks.

Problem

Standard MLLM training relies solely on next-token prediction with text supervision, which causes models to lean toward language comprehension and underutilize the rich visual perception signals needed for spatial/depth reasoning. Multi-encoder approaches that feed auxiliary visual features as extra inputs improve perception but introduce inference latency and require large training budgets. No prior work has directly optimized the quality of visual representations inside the LLM.

Method

VisPer-LM adds a predictive embedding distillation objective during the pre-training (PT) stage of LLaVA-style training. Three expert encoders provide target features: DINOv2-L from Depth Anything v2 (depth, E_depth), Swin-L from OneFormer (segmentation, E_seg), and CLIP-ViT-L from unCLIP-SD-2.1 (generation, E_gen). At selected middle LLM layers (D={8,20}, S={10,18}, G={12,20} for a 32-layer Llama3-8b), single-layer Perceiver Resampler embedding predictors map the LLM's hidden states to the target feature space via learnable queries. The embedding loss is a weighted sum of Smooth-L1 and InfoNCE (contrastive) losses. Special task tokens ⟨g⟩, ⟨d⟩, ⟨s⟩ derived from the predictors' learnable queries are appended to the image tokens in the input sequence, providing implicit visual chain-of-thought. The total PT loss is L_NTP + λ_depth·L_depth + λ_seg·L_seg + λ_gen·L_gen. During IFT and inference, only a single base vision encoder is used; the embedding predictors are dropped and the task tokens are frozen.

Key Contributions

  • First systematic probing study of LLM hidden-state visual representation quality in MLLMs; identifies a strong positive correlation (0.98) between depth probing cosine similarity and CV-Bench accuracy.
  • VisPer-LM: first method to inject visual perception knowledge from multiple expert encoders into the LLM's intermediate representations via predictive embedding distillation, with zero inference overhead vs. the single-encoder baseline.
  • Demonstrates that middle LLM layers (12–24 in Llama3-8b) are the optimal targets for embedding distillation, grounded in probing experiments.
  • Shows superiority over both single-encoder and multi-encoder input baselines, as well as encoder-level distillation approaches (RADIO-ViT-L), across perception-focused and general VQA benchmarks.

Results

  • VisPer-LM (CLIP-ConvNeXT-XXL + Llama3-8b) vs. LLaVA-1.5: +8.7% Depth, +5.6% Distance, +4.5% Relation on CV-Bench; +2.5% average.
  • VisPer-LM outperforms multi-encoder feature-concatenation and token-concatenation baselines by up to 0.9% average on CV-Bench (Tab. 1).
  • Outperforms RADIO-ViT-L-based LLaVA-1.5 (encoder-level distillation baseline) by 2.0% average (57.3 vs. 55.1).
  • BLINK benchmark: VisPer-LM 58.8 avg vs. LLaVA-1.5 55.6, Cambrian-1 56.9 (Tab. 4).
  • General reasoning benchmarks (POPE, GQA, MMMU, VizWiz): VisPer-LM 60.7 avg vs. LLaVA-1.5 59.7 (Tab. 2), confirming no regression.
  • Inference throughput: 9.86 vs. 9.92 samples/sec (single-encoder), 5.32 (multi-encoder) — near-identical to single-encoder baseline (Tab. 8).
  • SigLIP-ViT-SO400M base encoder: +3.4% Depth, +8.9% Relation on CV-Bench (Tab. 9).

Limitations

  • Focuses on improving perceptual (depth/spatial) reasoning; does not target general visual reasoning improvements, relying on the hope that NTP preserves them.
  • Experiments limited to image modality; video (motion/temporal reasoning) not explored.
  • Only validated with LLMs up to 8B parameters (Llama3-8b, Phi3-mini); results at 70B+ scale are unknown.
  • Expert encoder selection (depth, seg, gen) is hand-curated; more general-purpose teachers (e.g., InternViT) are left as future work.
  • Probing setup and embedding loss design require non-trivial hyperparameter choices (layer sets D/S/G, loss weights) with moderate sensitivity.

Relevance to Vision-Language Models

VisPer-LM directly addresses a known weakness of CLIP-based VLMs: their features are language-aligned but lack fine-grained geometric and perceptual detail, which propagates into the LLM as impoverished spatial representations. The paper's core insight—that visual representation quality inside the LLM correlates with perception task performance and can be directly optimized—opens a new axis of MLLM training beyond scaling data or stacking encoders. For researchers tracking VLMs, this is an orthogonal and inference-efficient alternative to the dominant multi-encoder paradigm (e.g., Cambrian-1), with implications for embodied AI and robotics applications that require robust depth and spatial reasoning.