Skip to content

See Only When Needed: Context-Aware Attention Intervention for Mitigating Hallucinations in LVLMs

🕒 Published (v1): 2026-06-29 06:35 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

Snoek group; ECCV 2026; training-free hallucination mitigation without amplifying spurious regions

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CAI introduces a training-free, token- and layer-selective attention intervention that amplifies visual grounding only for high-entropy tokens in deep decoding layers, while leaving confident tokens and irrelevant regions untouched. It achieves state-of-the-art hallucination mitigation across LLaVA-1.5, InstructBLIP, and Qwen-VL on POPE, CHAIR, and MME, with minimal latency overhead (1.03×). The paper also provides a formal characterization as KL-minimal attention reweighting with bounded interference.

Problem

LVLMs hallucinate due to over-reliance on language priors and degraded visual grounding in deeper decoding layers. Prior training-free methods uniformly boost attention to all visual tokens, which amplifies irrelevant/spurious regions, introduces new errors, and harms fluency. The key gap is the lack of a selective intervention mechanism that strengthens vision only where and when needed.

Method

CAI operates at inference time with three components:

  1. Token-aware visual grounding — extracts the 0-th layer hidden state of the current token, computes its cosine similarity (via inner product after L2 normalization, sigmoid-scaled) with each visual patch token to produce token-specific relevance weights \(\mathbf{w}_t \in (0,1)^{N_v}\).

  2. Similarity-guided attention intervention — at each decoding step, multiplies the attention magnitude \(|\mathbf{A}_{t,i_s:i_e}|\) (from the \(t\)-th token to all \(N_v\) image tokens across heads) element-wise by \(\mathbf{w}_t\): \(\mathbf{A}_{t,i_s:i_e} \leftarrow \mathbf{A}_{t,i_s:i_e} + |\mathbf{A}_{t,i_s:i_e}| \odot \mathbf{w}_t\), amplifying attention proportionally to semantic relevance.

  3. Entropy- and depth-gated activation — intervention only applies when two conditions jointly hold: (i) layer depth \(l > l_s\) (deeper layers where visual signal decays geometrically, proved in Theorem 3), and (ii) predictive entropy \(-\sum p(h_t^l)\log p(h_t^l) > \gamma\) (high-uncertainty tokens at risk of hallucination). Confident tokens and shallow layers remain unmodified.

Contrastive decoding (Eq. 7) is optional as an auxiliary bias-suppression module; CAI's attention-only variant (CAI\(^\dagger\), \(\lambda=1\)) already yields significant gains.

Key Contributions

  • Two-axis selective intervention — token-specific where (localize via early-layer similarity) and uncertainty/depth-gated when (activate only for high-entropy tokens in deep layers), enforcing a "see only when needed" principle.
  • Early-to-late grounding transfer — early-layer representations supply relevance signals; intervention is applied only in deeper layers where visual grounding naturally degrades, avoiding uniform amplification.
  • Principled theoretical characterization — proves CAI is the KL-minimal reweighting that maximizes expected token-image relevance (Theorem 1); guarantees NLL reduction for high-entropy tokens under small tilts (Theorem 2); shows visual signal decays exponentially with depth justifying late-layer intervention (Theorem 3); and bounds total-variation perturbation under inactive gates (Theorem 4).

Results

  • POPE — CAI achieves best accuracy and F1 across all 9 dataset×backbone settings (MS-COCO, A-OKVQA, GQA on LLaVA-1.5, InstructBLIP, Qwen-VL), including the hardest adversarial split (e.g., LLaVA-1.5 adversarial: 77.40 Acc vs. next-best ONLY 70.47).
  • CHAIR — Reduces CHAIR\(_S\) from 26.0→17.8 and CHAIR\(_I\) from 8.8→6.9 on LLaVA-1.5 (max 64 tokens); improvements hold at max 128 tokens (39.6 CHAIR\(_S\) vs. next-best 48.4).
  • MME — Highest total scores: 660.00 (LLaVA-1.5, +31.67 over ONLY), 510.00 (InstructBLIP, +43.34), 675.00 (Qwen-VL, +25.00), with consistent gains in object existence, count, position, and color subtasks.
  • Ablation (CAI\(^\dagger\), attention-only) — Already outperforms Regular baseline (POPE +2.32, CHAIR -13.0, MME +20.00) with negligible overhead (1.03× latency, 0.97× throughput), whereas PAI\(^\dagger\) (uniform amplification) yields flat or negative results.
  • Efficiency — Full CAI adds only 1.47× latency (131.33 ms/token) and 0.72× throughput relative to Regular, comparable to PAI (1.38×).

Limitations

  • Hyperparameters \(l_s\), \(\gamma\), \(\lambda\) require grid search per backbone; no adaptive/automatic tuning scheme.
  • Contrastive decoding still requires a text-only forward pass, adding some overhead.
  • The similarity signal is drawn only from layer 0; representations at that layer may not be optimal for all architectures.
  • Theoretical analysis relies on a linearized logit approximation (Theorem 2) and geometric visual decay model (Theorem 3) that may not hold perfectly in practice.
  • Evaluated only on object hallucination benchmarks; hallucination of attributes, relations, and actions is not explicitly studied.

Relevance to Vision-Language Models

This work directly addresses a central reliability problem in LVLMs: object hallucination arising from language-prior dominance. The key insight — that not all tokens need visual reinforcement equally, and that uniform boosting can cause new errors — reframes the training-free intervention paradigm from "amplify everything" to "amplify selectively." For VLM researchers, the formal KL-minimality proof and depth-entropy gating provide a principled template for designing future non-interference interventions, and the code release enables immediate integration into existing LVLM pipelines.