Skip to content

Visuals Lie, Consistency Speaks: Disentangling Spatial Attention from Reliability in Vision-Language Models

🕒 Published (v1): 2026-06-16 00:58 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; challenges Attention-Confidence Assumption; exposes core VLM reliability blind spot

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

The paper challenges the widely-held "Attention-Confidence Assumption"—that sharp visual attention predicts VLM reliability—through a systematic cross-family analysis of LLaVA-1.5, PaliGemma, and Qwen2-VL. Spatial attention metrics show near-zero correlation with correctness (\(R \approx 0.001\)), while hidden-state probes achieve AUROC > 0.95 and self-consistency achieves \(R = 0.429\). Architectural divergence is stark: LLaVA encodes reliability in a fragile late-stage bottleneck, whereas PaliGemma and Qwen2-VL distribute it globally.

Problem

Interpretability practice in VLMs conflates visual attention quality with output reliability—if a model attends to the right region, it is assumed to "know" the answer. This assumption has never been rigorously tested across model families with causal interventions. The gap is a principled method for predicting VLM hallucination at inference time without relying on untrustworthy attention heatmaps.

Method

The VLM Reliability Probe (VRP) instruments three stages of generation:

  1. Structural metrics: Cross-attention maps \(A^{(l,h)} \in \mathbb{R}^{T \times S}\) are aggregated over heads and answer-token positions to compute cluster count \(C_k\) and spatial entropy \(H_s\), plus layer-wise evolution \(\Delta H_s\).
  2. Mechanistic probes: Hidden states \(h^{(\ell)} \in \mathbb{R}^d\) are analyzed via Logit Lens (truth margin \(\Delta M_l\) = logit difference between correct and top-incorrect token), dense MLP probes, and sparse \(L_1\)-regularized logistic probes (\(\lambda=0.1\)) trained on activations to predict correctness.
  3. Behavioral metrics: \(K=10\) outputs are sampled; self-consistency (SC) is the majority-vote agreement rate.

Causal interventions ablate top-\(k\) predictive neurons and scale up to \(>50\%\) of hidden dimensions to test architectural robustness. Evaluated on POPE (adversarial, \(n=1000\)), LLaVA-Bench, custom counting/spatial tasks, VQA v2, and TextVQA.

Key Contributions

  • Cluster Failure: \(R(C_k, y) = 0.001\) (95% CI: \([-0.034, 0.036]\)), \(R(H_s, y) = -0.012\) on a pooled 3,090-sample structural-analysis set; even a supervised XGBoost-RF ensemble on 11 attention features reaches only 52–55% accuracy.
  • Symbolic Detachment: Mechanistic analysis reveals "Early Locking" (LLaVA attention sharpens at Layer 2, \(\Delta H_s \approx -2.5\), then diffuses at the final layer) vs. "Cyclical Refinement" (Qwen2-VL) and "Steady Decay" (PaliGemma)—explaining why attention maps are decorrelated from final output.
  • Reliability hierarchy: Hidden-state probes (AUROC > 0.95) > self-consistency (\(R = 0.429\), \(10\times\) cost) > logit entropy (AUROC \(\approx 0.50\)) > spatial attention (AUROC \(\approx 0.50\)).
  • Architectural divergence via causal ablation: LLaVA loses 8.3 pp on object identification when only 5 top-predictive neurons are ablated; PaliGemma loses only 1.0% accuracy when 1,000 neurons (\(\sim 50\%\) of its 2048 hidden size) are destroyed; Qwen2-VL shows zero measurable degradation (\(\Delta \leq \pm 2.0\) pp) even when \(>55\%\) of its residual dimension (2,000/3,584 neurons) is ablated.
  • MLP dominance: MLP layers contribute 82.1% of truth-margin growth at peak layer in LLaVA, confirming reliability is a feature-processing rather than token-routing phenomenon.

Results

  • Spatial attention classifier: 52–55% accuracy across all families (near chance); architecture-specific deeper probe reaches AUROC 0.725 at best.
  • Hidden-state probes on POPE: LLaVA 0.956, PaliGemma 0.738, Qwen2-VL 0.971 AUROC.
  • Self-consistency (K=10): AUROC 0.78/0.81/0.79 for LLaVA/PaliGemma/Qwen2-VL; precision at SC=1 is 90.8% (95% CI: [88.4, 92.8]%).
  • Masking top 30% attended patches: LLaVA accuracy \(-8.2\) pp, PaliGemma \(-11.3\) pp—confirming attention is causally necessary for feature extraction but not predictive of uncertainty.
  • VQA v2 / TextVQA: probes outperform output confidence in 3 of 6 model-task comparisons; gains are architecture- and benchmark-dependent.

Limitations

  • Only three VLM families evaluated; generalization to decoder-only VLMs (e.g., InternVL, LLaMA-Vision) is untested.
  • Self-consistency requires \(K=10\) forward passes (\(10\times\) inference cost), limiting practical deployment.
  • Hidden-state probes require architecture-specific training and show degraded performance on PaliGemma (AUROC 0.738) and on out-of-distribution benchmarks (VQA v2/TextVQA vs. POPE).
  • Sparse neuron ablations are conducted on relatively small causal validation splits (\(n=100\)–200), limiting statistical power for fine-grained effect sizes.
  • The study does not propose a mechanism to improve reliability, only to predict it.

Relevance to Vision-Language Models

This work directly challenges one of the foundational assumptions in VLM interpretability—that attention maps are a proxy for grounding quality—with rigorous empirical evidence and causal intervention, which has broad implications for hallucination detection and trustworthy deployment. The identification of architecture-dependent reliability circuits (fragile late bottleneck in LLaVA vs. distributed global encoding in PaliGemma/Qwen2-VL) provides a concrete mechanistic lens for comparing VLM design choices. For practitioners building VLM reliability monitors, the finding that hidden-state probes (AUROC > 0.95) far outperform attention-based heuristics suggests a shift toward latent-space diagnostics. The "Symbolic Detachment" phenomenon—early visual feature locking followed by diffusion before output generation—also informs how future VLM architectures should be evaluated for the fidelity of vision-language integration.