Intervene-All-Paths: Unified Mitigation of LVLM Hallucinations across Alignment Formats¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

AllPath is a training-free, inference-time framework that mitigates hallucinations in Large Vision-Language Models (LVLMs) by jointly intervening on all three causal pathways—image-to-input-text, image-to-output-text, and text-to-text—via targeted attention head scaling. A key empirical finding is that LVLMs adaptively rely on different pathway combinations depending on the question–answer alignment format (e.g., binary vs. open-ended captioning), so single-path intervention methods fail to generalize across benchmark types. AllPath identifies critical heads per pathway with a single forward pass and outperforms all baselines across all tested benchmarks.

Problem¶

Existing hallucination mitigation methods for LVLMs intervene on only a single causal pathway—either from the image to output text (e.g., PAI) or from input text to output text (e.g., AD-HH). Because LVLMs dynamically utilize different pathway combinations based on question format, single-path methods excel on some benchmarks while degrading on others, failing to provide consistent hallucination reduction across diverse alignment formats (binary, MCQ, open-ended captioning).

Method¶

AllPath operates in two stages:

Head identification (single forward pass): Two lightweight scoring methods identify critical attention heads per pathway type:
Text-to-Text (T2T) Score: A refined Log Probability Increase (LPI) score measures each head's differential contribution to hallucinated vs. non-hallucinated token probabilities. For short-answer questions the first decoding step is analyzed; for open-ended questions, per-token hallucination labels are used.
Image-to-Text (I2T) Score: For tokens where objects first appear (to avoid text-carried context), the ratio of attention concentrated on semantically aligned image regions (for present objects) vs. diffuse attention over the full image (for absent objects) identifies heads that genuinely ground visual content.
Multi-path intervention: Top-ranked T2T and I2T heads are collected into suppression set \(Z^-\) (hallucination-promoting T2T heads) and amplification set \(Z^+\) (hallucination-suppressing T2T heads ∪ image-grounding I2T heads). During inference, MHA outputs are scaled by \(\gamma^- = 0\) or \(\gamma^+ = 2\) for heads in each set, with all others unchanged. The pathway selection (I2T input vs. output text heads) is adapted to question format (discriminative vs. generative).

Key Contributions¶

First multi-path hallucination intervention framework for LVLMs, covering image-to-input-text, image-to-output-text, and text-to-text causal paths simultaneously.
Two novel, single-pass head-probing methods (T2T Score via LPI and I2T Score via attention concentration ratio), substantially more efficient than prior zero-out or classifier-based approaches.
Empirical finding that T2T heads are highly correlated within the same alignment format (ρ ≈ 0.84) but decorrelate sharply across formats (ρ ≈ 0.12–0.16), implicating alignment-format dependence as a key axis of hallucination.
Empirical finding that image-to-input-text and image-to-output-text heads are functionally independent (ρ ≈ 0.44), motivating their separate identification and joint use.
New MCQ-POPE benchmark extending POPE to multiple-choice format for cross-format evaluation.

Results¶

POPE (COCO): AllPath achieves +2.1% F1 over the best baseline (PAI) on the random split; consistent gains across random/popular/adversarial splits (best baseline improves ≤0.9% F1).
MCQ-POPE (COCO): AllPath achieves +5.2% accuracy over the best prior method (AD-HH at 78.6%, AllPath at 80.5% on random).
CHAIR: AllPath reaches CS=26.6 / CI=7.2, vs. the best baseline PAI (CS=28.8 / CI=7.9); +2.6% F1 improvement, exceeding best baseline gain of 1.9%.
MME hallucination subset: AllPath scores 600.0 total vs. 576.7 for best baseline (ICD), without recomputing heads specifically for MME (using POPE heads directly), demonstrating cross-task generalization.
Ablation confirms both T2T and I2T head sets are individually beneficial but combined use yields the largest gains.
Method is hyperparameter-insensitive: accuracy on POPE random varies only 85.6–88.3 across a wide sweep.

Limitations¶

Primary experiments are conducted on LLaVA-v1.5-7B; results for Qwen-VL-Chat and Qwen2.5-VL-7B/72B are in the appendix only.
The MCQ-POPE benchmark is derived from POPE and shares its object-centric, binary-presence framing; broader question types (e.g., spatial reasoning, counting) are not directly evaluated.
Hyperparameters (\(\xi\), \(\zeta\), \(\gamma^+\), \(\gamma^-\)) are chosen per-format class (discriminative vs. generative) rather than per-benchmark, which may leave per-task performance sub-optimal.
The I2T head identification requires object-presence ground truth and region annotations (e.g., COCO masks) to define aligned regions \(M_r\), which may not generalize trivially to datasets without such annotations.
The hallucination detection used for open-ended T2T scoring (Alg. 2) relies on an external judge to label tokens, introducing a dependency on the quality of that oracle.

Relevance to Vision-Language Models¶

AllPath directly addresses a fundamental interpretability and reliability question in VLMs: where, mechanistically, do hallucinations originate within the transformer? By showing that hallucination is a multi-path phenomenon tied to alignment format, this work challenges the common single-axis framing (visual grounding vs. language prior) and provides actionable surgical interventions at the attention-head level. It advances the line of mechanistic VLM analysis (alongside AD-HH, PAI, VCD) toward a unified, format-aware view, which is critical as deployed VLMs serve increasingly diverse query types. The introduced MCQ-POPE benchmark also fills a gap for evaluating models under multiple-choice alignment, a prevalent real-world format.