Dismantling Pathological Shortcuts: A Causal Framework for Faithful LVLM Decoding

🕒 Published (v1): 2026-06-25 22:55 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML; causal root-cause of LVLM hallucination beyond attention heuristics

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LVLMs hallucinate not because of insufficient visual attention overall, but because specific attention heads ("risky mediators") structurally decouple from visual tokens at decision-critical decoding steps, creating a shortcut routed through language priors. Fox is a training-free inference-time framework that diagnoses these heads via visual attention entropy, severs the shortcut via numerical logit saturation (the do-operator), and reconciles faithfulness with fluency via conflict-gated decoding. It outperforms the prior best (SID) by up to 29.1% on CHAIR\(_I\) without retraining.

Problem

Existing hallucination mitigation assumes a quantitative deficit in visual attention (the "attention intensity assumption") and responds by globally amplifying visual signals or suppressing language priors. Empirically, hallucinated outputs show no statistically significant reduction in global visual attention magnitude (\(m_{V,\text{tail}}\) distributions largely overlap), so global intensity boosting fails. The actual failure is a dynamic structural misalignment: certain attention heads functionally disconnect from visual evidence precisely at content-bearing generation steps, routing predictions through system-instruction priors.

Method

Fox operates in three stages grounded in a Structural Causal Model (SCM):

1. Causal Diagnosis of Risky Mediators. Two temporal "decision-critical" query positions are identified: the multimodal handshake \(x_\text{last}\) (last prefix token, aggregates visual-linguistic context) and the autoregressive anchor \(y_{t-1}\) (proximal causal parent). At these positions, each head \((l, h)\) is scored by a joint risk metric: $\(S^{(l,h)} = m^{(l,h)}_\text{sys} \cdot H^{(l,h)}_\text{vis}\)$ where \(m^{(l,h)}_\text{sys}\) measures attention mass on system tokens \(I_\text{sys}\), and \(H^{(l,h)}_\text{vis}\) is the Shannon entropy of the locally re-normalized visual attention distribution \(\hat{A}^{(l,h)}\) — high entropy signals visual uncertainty. Heads in the top-\(k\) fraction of \(S\) are designated risky mediators \(H_R\) (AUC = 0.818 for distinguishing hallucinated trajectories).

2. Causal Intervention via Numerical Logit Saturation. The do-operator \(\text{do}(H_R := \text{noise})\) is implemented by applying a large negative bias \(\gamma\) to the pre-softmax logits of diagnosed heads at the critical steps: $\(\tilde{L}^{(l,h)} = \Pi_\text{dtype}(L^{(l,h)} - \gamma), \quad (l,h) \in H_R\)$ In finite-precision arithmetic, \(e^{x-\gamma} \to \epsilon_\text{mach}\), effectively zeroing those attention weights and forcing routing through stable visual mediators \(H_S\). Strong structural signals survive; spurious prior-anchored signals do not.

3. Conflict-Gated Cooperative Decoding. Two distributions are maintained: observational \(P_\text{obs}\) (unmodified) and interventional \(P_\text{do}\) (from modified heads). Their disagreement is measured by truncated Jensen-Shannon Divergence over high-probability vocabulary \(V_t\): $\(d_t = \text{JSD}(P_\text{obs}(\cdot|V_t) \| P_\text{do}(\cdot|V_t))\)$ Final logits: \(z_\text{final} = z_\text{obs} + \lambda_t \cdot z_\text{do}\), where \(\lambda_t = \alpha\) (fixed amplification) when \(d_t < \tau_{JS}\) (consensus), and \(\lambda_t = d_t\) (calibrated correction) when \(d_t \geq \tau_{JS}\) (conflict).

Key Contributions

• Challenges the attention intensity assumption with empirical evidence that hallucination is driven by dynamic structural misalignment, not global visual attention deficit.
• Identifies "risky mediators" — sparse attention heads that decouple from visual tokens at decision-critical steps — as the mechanistic source of pathological shortcuts.
• Proposes Fox, a training-free SCM-grounded framework using visual attention entropy as an unsupervised probe, numerical logit saturation as the do-operator, and conflict-gated fusion for faithfulness-fluency balance.
• Achieves SOTA across POPE, CHAIR, and MME on three LVLM backbones (LLaVA-1.5, InstructBLIP, Shikra) without modifying model weights.

Results

• CHAIR\(_I\) (LLaVA-1.5): 12.90 vs. SID 15.40 → 16.2% relative reduction; vs. baseline sampling 17.18.
• CHAIR\(_I\) (InstructBLIP): 11.98 vs. SID 16.89 → 29.1% relative reduction.
• CHAIR\(_S\) (LLaVA-1.5): 46.40 vs. SID 53.40; (InstructBLIP) 42.40 vs. SID 54.00.
• POPE Adversarial (LLaVA-1.5): Fox 81.93% vs. SID 81.10%, OPERA 81.13%, VCD 75.33%.
• MME (LLaVA-1.5): Total 613.33 vs. SID 600.00; Position subset 131.37 vs. 93.33 (baseline).
• Diagnosis-driven head selection consistently outperforms random-intervention baseline across all \(K\) ratios (POPE \(\Delta\)Mean Acc and \(\Delta\)Mean F1 both positive).

Limitations

• Hyperparameters \((k, \tau_{JS}, \alpha, \beta)\) require per-backbone tuning (different values for LLaVA-1.5 vs. InstructBLIP/Shikra).
• Evaluated only on three older LVLM architectures (LLaVA-1.5, Shikra, InstructBLIP); generalization to more recent instruction-tuned models (e.g., LLaVA-Next, InternVL, Qwen-VL) is not demonstrated.
• The numerical logit saturation mechanism relies on finite-precision behavior (\(\Pi_\text{dtype}\)), making the intervention's effect potentially architecture- and precision-dependent (e.g., bfloat16 vs. float16 vs. float32).
• No computational overhead analysis provided relative to beam-search baselines like OPERA.
• The SCM formulation assumes the pathological shortcut is mediated solely via \(X_\text{sys}\); other prior sources (textual history \(X_\text{txt}\)) are not analyzed.

Relevance to Vision-Language Models

Fox directly addresses a core failure mode of LVLMs — object hallucination — by providing mechanistic interpretability of which attention heads cause grounding failures and when. The shift from global attention statistics to head-level structural causal analysis represents a more principled diagnostic paradigm than prior contrastive decoding or attention-reweighting methods, with implications for any VLM inference pipeline requiring visual faithfulness. The training-free, model-agnostic design makes it immediately applicable as a decoding wrapper. The causal intervention framework also opens a path toward runtime monitoring of hallucination risk, relevant to safety-critical VLM deployments (medical imaging, embodied AI).