Mitigating Object Hallucinations in LVLMs via Attention Imbalance Rectification

🕒 Published (v1): 2026-03-25 08:06 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; attention imbalance rectification directly mitigates object hallucination

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Object hallucinations in LVLMs arise from imbalanced attention allocation—both across modalities (vision vs. language) and within a sequence (token-level). This paper introduces Attention Imbalance Rectification (AIR), a training-free, decoding-time intervention that reallocates cross-modal attention and regularizes attention variance to suppress hallucinations, achieving up to 35.1% reduction in sentence-level hallucination rate across four LVLMs.

Problem

LVLMs systematically hallucinate non-existent objects, but the mechanistic cause has lacked precise characterization. Existing mitigations—instruction fine-tuning, post-hoc correction, and contrastive decoding—are either costly, add inference overhead, or generalize poorly. The root cause connecting attention behavior to hallucination has not been rigorously quantified.

Method

The paper introduces two complementary measures of attention imbalance:

• Modality-Wise Attention Imbalance (MAI): \(\mathrm{MAI}(M_p, M_q) := A_{M_p} / A_{M_q}\), the ratio of total attention mass received by two modalities. Hallucination-sensitive heads exhibit MAI ≈ 5.1 (strongly text-biased) vs. 1.5 for hallucination-insensitive heads.
• Token-Wise Attention Imbalance (TAI): \(\mathrm{TAI}(x_j) := \frac{A(x_j)}{\sum_k A(x_k)} \cdot \frac{\sum_k I(x_k; x_{i+1} \mid x_{\le i})}{I(x_j; x_{i+1} \mid x_{\le i})}\), the ratio of attention share to information contribution (via conditional mutual information). Over-attended tokens (\(\mathrm{TAI} \gg 1\)) consistently precede hallucinated content within 15 tokens.

AIR intervenes at decoding time with two components:

1. Modality-Balanced Attention Reallocation: For each hallucination-sensitive head \((\ell, h) \in \mathcal{H}_\text{sens}\), when cumulative text-token attention \(V^{\text{text}}_{(\ell,h)} > \tau_\text{text}\), suppress text attention by \(\lambda \in [0,1]\) and amplify visual attention by \(\gamma > 1\): \(A'^{(\ell,h)}_{ij} = \lambda A^{(\ell,h)}_{ij}\) for \(j \in \mathcal{T}\), \(\gamma A^{(\ell,h)}_{ij}\) for \(j \in \mathcal{V}\).

2. Variance-Constrained Projection Regularization: Adaptively rescales \(W_\text{QK}\) by spectral energy, then applies zero-trace projection \(\hat{A} = A - \frac{\mathrm{tr}(A)}{L}I\), Frobenius-norm-preserving rescaling, and shrinkage toward the mean: \(A^* = (1-\beta)\tilde{A} + \beta\,\mathrm{mean}(\tilde{A})\mathbf{1}\).

Hallucination-sensitive heads are identified offline via erasure-based attribution (removing each head and measuring change in hallucination probability). The method incurs no additional inference overhead.

Key Contributions

• Formal definition of attention imbalance (MAI and TAI) as a quantitative, interpretable causal indicator of object hallucination in LVLMs.
• Empirical finding that hallucination-sensitive heads inherit text-biased attention patterns from their base language model (cosine similarity 0.81 vs. 0.69 for insensitive heads with Vicuna-7B).
• AIR: a training-free, decoding-time method with no added inference overhead, operating on pre-identified hallucination-sensitive attention heads.
• Demonstration of consistent gains across 4 LVLMs (LLaVA-1.5, MiniGPT-4, InstructBLIP, Shikra) on 3 benchmarks (CHAIR, POPE, MM-Vet) against 7 baselines.

Results

• CHAIR (max 256 tokens): AIR reduces \(C_S\) / \(C_I\) by up to 35.1% / 22.6% (MiniGPT-4: 32.8→21.3 / 11.5→8.9) over the second-best baseline (AD-HH); LLaVA-1.5: 35.2→28.8 (\(C_S\)), 8.8→8.6 (\(C_I\)).
• POPE: AIR achieves highest average accuracy across all four LVLMs on Adversarial, Popular, and Random subsets.
• MM-Vet (general capability): AIR improves general LVLM capability by up to 15.9% over baselines.
• Outperforms all 7 baselines (Greedy, FarSight, VCD, DoLA, HALC, OPERA, AD-HH) consistently under both 256 and 64 max-new-token settings.

Limitations

• Hallucination-sensitive head identification relies on an erasure-based attribution pass that must be run per model (offline cost not detailed in the main text).
• Hyperparameters (\(\lambda\), \(\gamma\), \(\beta\), \(\xi\), \(\tau_\text{text}\)) require tuning; sensitivity analysis is deferred to the appendix.
• Evaluated on four relatively older open-source LVLMs; applicability to more recent or closed-source models (e.g., LLaVA-1.6, InternVL, GPT-4V) is not demonstrated.
• The TAI formulation depends on conditional mutual information estimates that may be noisy or expensive to compute accurately.
• The snowball/propagation hallucination phenomenon is observed but not explicitly modeled or suppressed.

Relevance to Vision-Language Models

This work provides a principled, mechanistic account of object hallucination in LVLMs by directly linking attention head behavior to failure modes—grounding the diagnosis in both modality-level and token-level statistics rather than heuristics. For researchers tracking VLMs, AIR is notable as a zero-training, zero-overhead decoding intervention, contrasting with the dominant contrastive decoding and fine-tuning paradigms. The finding that hallucination-sensitive heads inherit LLM-style text-dominant attention patterns offers a new lens for understanding how vision-language fusion fails, with implications for architecture design (e.g., how cross-modal connectors and attention heads should be trained). The dual-axis framework (MAI + TAI) is broadly applicable to any transformer-based LVLM and could serve as a diagnostic tool for future hallucination research.