Skip to content

A Comprehensive Information-Decomposition Analysis of Large Vision-Language Models

🕒 Published (v1): 2026-03-31 12:32 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; information-decomposition reveals if VLMs use true multimodal fusion or shortcuts

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper applies partial information decomposition (PID) to quantitatively characterize how 26 large vision-language models (LVLMs) use visual vs. textual information, decomposing model decisions into redundancy \(R\), vision uniqueness \(U_1\), language uniqueness \(U_2\), and synergy \(S\). The analysis spans cross-model breadth, layer-wise depth, and training-time dynamics, revealing two task regimes and two stable family-level strategies. Visual instruction tuning is identified as the stage where cross-modal synergy \(S\) is learned.

Problem

Accuracy metrics alone cannot reveal whether LVLM success stems from genuine multimodal fusion or from language priors. Existing interpretability approaches either focus on a single modality in isolation or use ad hoc metrics lacking theoretical grounding. No comprehensive, quantitative framework existed for simultaneously profiling information composition, layer-wise flow, and training-time evolution across a diverse set of modern LVLMs.

Method

The framework treats vision embeddings \(X_1\) (mean-pooled visual tokens) and language embeddings \(X_2\) (mean-pooled text tokens) as PID source variables, with the model's output \(Y\) over a finite multiple-choice answer set as the target. PID decomposes total mutual information \(I(X_1, X_2; Y)\) into four non-negative atoms via:

\[R = \max_{Q \in \Delta_P} I_Q(X_1; X_2; Y), \quad U_i = \min_{Q \in \Delta_P} I_Q(X_i; Y | X_{-i}), \quad S = I(X_1, X_2; Y) - \min_{Q \in \Delta_P} I_Q(X_1, X_2; Y)\]

The BATCH estimator (Liang et al., 2023) is adapted with three key modifications for LVLMs: (1) unimodal conditioning via Gaussian noise masking \(\mathcal{N}(\mu, \text{diag}(\sigma^2))\) of the suppressed modality's embedding sequence; (2) confidence thresholding that replaces low-confidence renormalized distributions with uniform \(\mathcal{U}(K)\) to prevent spurious PID structure; (3) soft aggregation of \(P(Y)\) by averaging predictive distributions rather than argmax frequencies. The pipeline is model-agnostic and requires no architectural changes. It is applied to 26 models (0.5B–90B) from 11 families across 4 MC-VQA benchmarks (MMBench, POPE, Reefknot, PMC-VQA), plus layer-wise logit-lens analysis and training-checkpoint analysis of LLaVA-1.5.

Key Contributions

  • A model-agnostic PID estimation pipeline for LVLMs operating on MC-VQA outputs, with three adaptations (noise masking, confidence thresholding, soft aggregation) enabling robust estimation on modern architectures.
  • Identification of two task regimes: synergy-driven (MMBench, POPE) where \(S\) strongly predicts accuracy (\(\rho \approx 0.75\), \(p < 0.001\)) and knowledge-driven (Reefknot, PMC-VQA) where \(U_2\) becomes more predictive.
  • Discovery of two stable, family-level information strategies: fusion-centric families (InternVL2.5/3, Qwen2/2.5-VL) with high \(S\) and low \(U_2\) vs. language-centric families (Gemma3, Cambrian) with low \(S\) and high \(U_2\), consistent across task regimes.
  • A consistent three-phase layer-wise pattern in how PID components evolve across transformer depth.
  • Identification of visual instruction tuning (Stage 2, MLP + LM fine-tuning) as the key training stage where synergy \(S\) is unlocked; alignment pretraining alone does not produce it.
  • Intervention-based validation: image-removal accuracy drop \(D_\text{vision}\) correlates with \(S\) on synergy-driven tasks (\(\rho = 0.809/0.744\) for MMBench/POPE, both \(p < 0.001\)).

Results

  • On synergy-driven benchmarks (MMBench, POPE): \(S\) is the strongest positive correlate of accuracy (\(\rho = 0.750\)/\(0.742\), \(p < 0.001\)), outperforming total mutual information \(I(X_1, X_2; Y)\) (\(\rho = 0.632\)/\(0.157\)).
  • On knowledge-driven benchmarks (Reefknot, PMC-VQA): \(U_2\) becomes significant (PMC-VQA: \(\rho = 0.406\), \(p = 0.040\)); accuracy is 20–30% lower than synergy-driven tasks, suggesting \(S\) cannot compensate for missing domain knowledge.
  • Family-level \(S\)/\(U_2\) positions persist across both task regimes—family strategy is stable, not task-contingent.
  • Scaling within families does not systematically increase \(U_2\); fusion-centric families maintain or increase \(S\) with scale.
  • Visual instruction tuning checkpoints show a sharp rise in \(S\); alignment pretraining checkpoints show near-zero \(S\) despite improved \(U_1\) alignment.

Limitations

  • Framework is restricted to MC-VQA tasks due to BATCH estimator's requirement for a finite, discrete output space \(Y\); open-ended generation tasks are excluded.
  • Unimodal conditioning via noise masking is an approximation; residual cross-modal signal may remain after masking, potentially underestimating or overestimating \(U_1\)/\(U_2\).
  • PID estimation requires partitioning datasets 3:1 for BATCH training/testing, reducing effective evaluation set size.
  • Layer-wise and training-dynamics analyses focus on a limited set of representative models (InternVL3, Qwen2.5-VL, LLaVA-1.5); generalizability to other families is assumed but not demonstrated.
  • Outlier families (InstructBLIP, Fuyu) are omitted from family-strategy figures without full quantitative explanation.

Relevance to Vision-Language Models

This work provides the first comprehensive information-theoretic audit of modern LVLMs at scale, moving beyond accuracy-only benchmarking to reveal how models process multimodal inputs. The distinction between fusion-centric and language-centric family strategies has direct implications for model selection and architecture design: practitioners using domain-specific tasks (e.g., medical VQA) should favor models with high \(U_2\) capacity rather than optimizing purely for fusion. The finding that visual instruction tuning—not alignment pretraining—is where synergy \(S\) emerges offers a principled target for training recipe improvements and explains why instruction-tuned models generalize better on vision-heavy tasks. The PID lens also provides a complementary diagnostic to attention and probing analyses, enabling identification of modality-collapse failure modes without requiring ground-truth labels.