Visuals Lie, Consistency Speaks: Disentangling Spatial Attention from Reliability in Vision-Language Models¶

🕒 Published (v1): 2026-06-16 00:58 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; debunks attention-confidence assumption critical for reliable VLM agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

The paper challenges the widely-held "Attention-Confidence Assumption"—that sharp visual attention predicts VLM reliability—through a systematic cross-family analysis of LLaVA-1.5, PaliGemma, and Qwen2-VL. Spatial attention metrics show near-zero correlation with correctness (\(R \approx 0.001\)), while hidden-state probes achieve AUROC > 0.95 and self-consistency achieves \(R = 0.429\). Architectural divergence is stark: LLaVA encodes reliability in a fragile late-stage bottleneck, whereas PaliGemma and Qwen2-VL distribute it globally.

Problem¶

Interpretability practice in VLMs conflates visual attention quality with output reliability—if a model attends to the right region, it is assumed to "know" the answer. This assumption has never been rigorously tested across model families with causal interventions. The gap is a principled method for predicting VLM hallucination at inference time without relying on untrustworthy attention heatmaps.

Method¶

The VLM Reliability Probe (VRP) instruments three stages of generation:

Structural metrics: Cross-attention maps \(A^{(l,h)} \in \mathbb{R}^{T \times S}\) are aggregated over heads and answer-token positions to compute cluster count \(C_k\) and spatial entropy \(H_s\), plus layer-wise evolution \(\Delta H_s\).
Mechanistic probes: Hidden states \(h^{(\ell)} \in \mathbb{R}^d\) are analyzed via Logit Lens (truth margin \(\Delta M_l\) = logit difference between correct and top-incorrect token), dense MLP probes, and sparse \(L_1\)-regularized logistic probes (\(\lambda=0.1\)) trained on activations to predict correctness.
Behavioral metrics: \(K=10\) outputs are sampled; self-consistency (SC) is the majority-vote agreement rate.

Causal interventions ablate top-\(k\) predictive neurons and scale up to \(>50\%\) of hidden dimensions to test architectural robustness. Evaluated on POPE (adversarial, \(n=1000\)), LLaVA-Bench, custom counting/spatial tasks, VQA v2, and TextVQA.

Key Contributions¶

Cluster Failure: \(R(C_k, y) = 0.001\) (95% CI: \([-0.034, 0.036]\)), \(R(H_s, y) = -0.012\) on a pooled 3,090-sample structural-analysis set; even a supervised XGBoost-RF ensemble on 11 attention features reaches only 52–55% accuracy.
Symbolic Detachment: Mechanistic analysis reveals "Early Locking" (LLaVA attention sharpens at Layer 2, \(\Delta H_s \approx -2.5\), then diffuses at the final layer) vs. "Cyclical Refinement" (Qwen2-VL) and "Steady Decay" (PaliGemma)—explaining why attention maps are decorrelated from final output.
Reliability hierarchy: Hidden-state probes (AUROC > 0.95) > self-consistency (\(R = 0.429\), \(10\times\) cost) > logit entropy (AUROC \(\approx 0.50\)) > spatial attention (AUROC \(\approx 0.50\)).
Architectural divergence via causal ablation: LLaVA loses 8.3 pp on object identification when only 5 top-predictive neurons are ablated; PaliGemma loses only 1.0% accuracy when 1,000 neurons (\(\sim 50\%\) of its 2048 hidden size) are destroyed; Qwen2-VL shows zero measurable degradation (\(\Delta \leq \pm 2.0\) pp) even when \(>55\%\) of its residual dimension (2,000/3,584 neurons) is ablated.
MLP dominance: MLP layers contribute 82.1% of truth-margin growth at peak layer in LLaVA, confirming reliability is a feature-processing rather than token-routing phenomenon.

Results¶

Spatial attention classifier: 52–55% accuracy across all families (near chance); architecture-specific deeper probe reaches AUROC 0.725 at best.
Hidden-state probes on POPE: LLaVA 0.956, PaliGemma 0.738, Qwen2-VL 0.971 AUROC.
Self-consistency (K=10): AUROC 0.78/0.81/0.79 for LLaVA/PaliGemma/Qwen2-VL; precision at SC=1 is 90.8% (95% CI: [88.4, 92.8]%).
Masking top 30% attended patches: LLaVA accuracy \(-8.2\) pp, PaliGemma \(-11.3\) pp—confirming attention is causally necessary for feature extraction but not predictive of uncertainty.
VQA v2 / TextVQA: probes outperform output confidence in 3 of 6 model-task comparisons; gains are architecture- and benchmark-dependent.

Limitations¶

Only three VLM families evaluated; generalization to decoder-only VLMs (e.g., InternVL, LLaMA-Vision) is untested.
Self-consistency requires \(K=10\) forward passes (\(10\times\) inference cost), limiting practical deployment.
Hidden-state probes require architecture-specific training and show degraded performance on PaliGemma (AUROC 0.738) and on out-of-distribution benchmarks (VQA v2/TextVQA vs. POPE).
Sparse neuron ablations are conducted on relatively small causal validation splits (\(n=100\)–200), limiting statistical power for fine-grained effect sizes.
The study does not propose a mechanism to improve reliability, only to predict it.

Relevance to Agentic AI / LLM Agents¶

Agentic systems increasingly rely on VLMs as perception modules — deciding what to act on, when to ask for clarification, and when to abort a subtask — making reliable confidence estimation a core requirement rather than a nice-to-have. This paper directly undermines the common practice of using attention visualizations as a trust signal in such pipelines, showing they carry essentially no predictive value about correctness. Instead, it surfaces two practically deployable alternatives for agents: hidden-state probes (high accuracy but requiring fine-tuning access) and self-consistency sampling (AUROC-level signal with no model internals needed), which maps cleanly onto the "sample-then-verify" patterns already common in tool-calling and multi-agent reasoning loops. The architectural finding — that reliability is encoded in a brittle late bottleneck in LLaVA but distributed globally in PaliGemma and Qwen2-VL — gives agent builders concrete guidance on which backbone is more robustly self-aware under uncertainty.