Skip to content

Information-Regularized Attention for Visual-Centric Reasoning

🕒 Published (v1): 2026-07-01 04:45 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV; tackles hallucination, grounding, forgetting via attention regularization — core VLM reliability

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

IRA (Information-Regularized Attention) is a stochastic attention mechanism that injects learnable Gaussian noise into the value states of visual tokens during VLM fine-tuning, framing it as amortized variational inference layer-by-layer. It explicitly regularizes visual representations to suppress attention sink and reduce hallucination, rather than modifying attention weights post-hoc. Applied as a drop-in replacement for the SFT stage, IRA consistently outperforms standard supervised fine-tuning across general VQA, robustness, and OOD generalization benchmarks.

Problem

Standard next-token prediction SFT optimizes VLMs to maximize \(I(h; y)\) but does not discard irrelevant nuisance information \(I(h; x | y)\), causing visual embeddings to absorb redundant or spurious signals. This manifests as attention sink (attention collapsing onto semantically uninformative tokens), object hallucination, weak visual grounding, and catastrophic forgetting after full-parameter instruction tuning. Prior mitigation strategies (gating, attention weight redistribution) operate on attention outputs, leaving the intermediate visual representations unregulated.

Method

IRA re-frames each transformer layer as performing data-dependent amortized variational inference over the visual value states \(v\). A lightweight posterior sampler (one Linear(d, d+1) and one Embedding(H, d) per layer) parameterizes:

\[q_{\theta,\phi}(z \mid v) = \mathcal{N}\!\left(v + \Delta_\phi(v),\; \sigma_q^2 I\right)\]

with a stop-gradient prior \(p_\lambda(z \mid \lfloor v \rfloor) = \mathcal{N}(\lfloor v \rfloor, \sigma_p^2 I)\) to prevent KL collapse. The reparameterization trick yields \(z = v + \Delta_\phi(v) + \sigma_q \cdot \epsilon\), \(\epsilon \sim \mathcal{N}(0, I)\), replacing deterministic value states with stochastic ones only for visual tokens.

To avoid uniform over-regularization, an uncertainty-conditioned weight \(g_i = H \cdot (1 - a_i)\) is computed per visual token, where \(a_i\) is the average cross-modal attention mass assigned to token \(i\) and \(H\) is the normalized entropy of that attention distribution. Tokens receiving low attention under high uncertainty receive stronger regularization; tokens that carry high-fidelity visual signal are left relatively intact. The KL coefficient \(\beta\) is cosine-scheduled from 0 to \(\beta_{\max} = 10^{-4}\) over the first 50% of training steps. At inference, stochasticity is removed: \(z = v + \Delta_\phi(v)\).

Key Contributions

  • IRA: a stochastic, information-regularized attention mechanism that controls visual information flow at the value-state level during full-parameter SFT, with negligible parameter overhead.
  • Empirical demonstration that IRA produces smoother layer-wise curvature trajectories and suppresses attention sink across all layers, indicating more stable visual representation geometry.
  • Identification of a correlation between representational curvature (nonlinearity of visual embedding trajectories across transformer depth) and attention sink severity, offering a new diagnostic axis for VLM reliability.
  • Generalization to OOD settings (multi-image and video) after training exclusively on single-image data.

Results

All comparisons are IRA vs. SFT baseline (same pre-training checkpoint, same 3.2M instruction-tuning data):

  • InternVL2: MME 1598 (SFT) → 1792 (IRA); MMMU-Pro 26.0 → 30.1; MMMU 42.1 → 45.1; MME-RealWorld 31.2 → 37.6.
  • InternVL2.5: MME 1669 (SFT) → 1981 (IRA, from table); MMMU-Pro 26.7 → 30.4; MMMU 41.3 → 46.4; MME-RW 29.5 → 40.4.
  • LLaVA-OneVision: MME 1530 (SFT) → 2030 (IRA); MMMU-Pro 20.5 → 27.4; MMMU 36.3 → 44.4; MMStar 30.6 → 57.9; OK-VQA 36.8 → 75.9; TextVQA 37.6 → 79.9.
  • Robustness (LLaVA-OV): POPE 83.9 → 86.4; MuirBench (multi-image OOD) 33.6 → 35.7; MVBench (video OOD) 52.3 → 63.9.
  • Robustness (InternVL2.5): POPE 87.0 → 87.5; VLM-Blind 5.8 → 19.0; EmbSpatial 24.6 → 42.9.
  • Largest gains consistently on knowledge-intensive (OK-VQA) and STEM-reasoning benchmarks; text-intensive tasks (TextVQA, DocVQA, ChartQA) also improve, showing the uncertainty-conditioned weighting avoids over-regularizing high-fidelity visual signals.

Limitations

  • IRA is applied only at the SFT stage; whether it transfers gains to preference optimization or RLHF-style post-training is not studied.
  • Evaluated on single-image instruction tuning data (3.2M samples); multi-image and video settings are treated as OOD holdout only, not training targets.
  • The uncertainty-conditioned weight \(g_i\) is only used during training; inference collapses to a deterministic mean shift \(\Delta_\phi(v)\), so any test-time calibration benefit from stochasticity is lost.
  • Curvature analysis is presented qualitatively/visually (Fig. 4 truncated in text); no formal statistical validation of the curvature–sink correlation is given.
  • The KL schedule hyperparameters (\(\beta_{\max}\), warmup fraction \(k\)) require tuning per architecture.

Relevance to Vision-Language Models

IRA directly addresses two of the most persistent failure modes in VLMs—hallucination and attention sink—through a representation-level intervention rather than data curation or post-hoc attention redistribution, making it complementary to existing instruction-tuning and preference-optimization pipelines. The information-bottleneck framing provides a principled justification for why deterministic SFT underspecifies visual embeddings, a mechanistic insight applicable across VLM architectures. The empirical finding that smoother layer-wise curvature of visual token trajectories correlates with reduced hallucination opens a new geometric lens for diagnosing VLM reliability. For researchers tracking VLMs, IRA represents a lightweight, modular training-time intervention that can be layered on top of existing pre-training pipelines without architectural redesign.