Seeing but Not Believing: Probing the Disconnect Between Visual Attention and Answer Correctness in VLMs

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; probes disconnect between visual attention and answer correctness in VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLMs often attend to the correct visual evidence in their deep layers yet still produce wrong answers — a phenomenon the authors call "seeing but not believing." They propose VEA (Visual Evidence Augmentation), a training-free inference-time method that extracts deep-layer attention maps, denoises them, and overlays them as a highlight mask on the input image, consistently improving VQA accuracy across model families.

Problem

Prior work attributes VLM failures on visual questions to insufficient attention to image tokens overall, but this conflates perception with utilization. The core gap is: when a VLM answers incorrectly, has it failed to perceive the evidence, or has it perceived but failed to leverage it? Existing interventions do not separate these failure modes, and few analyze layer-wise internal attention dynamics to locate where the bottleneck actually lies.

Method

The paper first conducts a mechanistic analysis using Relative Attention Per Token (RAPT) — the ratio of per-token section-average attention to the input-average — across all transformer layers, and uses AUROC against human-annotated evidence bounding boxes (from VisualCOT) to score each layer's evidence attribution quality.

VEA (Visual Evidence Augmentation) proceeds in three steps: 1. Layer profiling (one-time): Run a small diagnostic subset (~100 examples) through the model; compute AUROC per layer against ground-truth evidence token labels; select the top 10% of layers as \(L_\text{VG}\). 2. Inference-time evidence attribution: Perform a single-token forward pass, average patch-level visual attention across \(L_\text{VG}\): $\(e_i = \frac{1}{|L_\text{VG}|} \sum_{\ell \in L_\text{VG}} \bar{a}_i^{(\ell)}\)$ Then apply neighborhood-based denoising to suppress attention-sink artifacts (a patch is replaced by its local \(3\times3\) mean if it exceeds \(\lambda=10\times\) the neighborhood maximum), followed by Gaussian smoothing. 3. Image augmentation: Blend the refined mask \(\tilde{e} \in \mathbb{R}^{H\times W}\) with the original image to dim low-evidence regions: $\(\hat{I}_{i,j,c} = \left(\alpha + (1-\alpha)\tilde{e}_{i,j}\right) I_{i,j,c}\)$ The augmented image \(\hat{I}\) replaces the original in a second full forward pass for answer generation. No gradients, no retraining.

Key Contributions

• Systematic layer-wise analysis showing shallow layers are text-focused and deep layers sparsely ground to localized evidence regions, with RAPT and AUROC measurements to quantify this.
• Empirical demonstration that the "seeing but not believing" phenomenon is widespread across LLaVA, Qwen, Gemma, and InternVL families — deep layers attend to correct evidence even when the final answer is wrong.
• VEA: a training-free, architecture-agnostic inference-time augmentation that uses the model's own deep-layer attention to highlight evidence in the input image.
• Two-part explanation for the disconnect: textual dominance from LLM backbone imbalance, and visual context under-utilization analogous to RAG context neglect.

Results

• VEA achieves an average +5.67 Exact Match and +6.83 Token F1 over base models across 8 VLMs on 4 VQA datasets (InfoVQA, DocVQA, SROIE, TextVQA).
• Maximum gains: +11.1 Exact Match (LLaVA-7B) and +17.3 Token F1.
• VEA achieves lowest average rank (1.12 EM / 1.22 F1) among all methods, outperforming baselines: Instructioning (INST), CGR (chain-of-visual-grounding), VAR (final-layer binary mask), and AGLA (GradCAM-based ensemble).
• AGLA is the strongest baseline; VEA surpasses it on all 4 model families.
• Gains are largest on smaller models (LLaVA-7B, Gemma-4B), but consistent across scales up to 32B.

Limitations

• Layer profiling requires ~100 annotated evidence-region examples (VisualCOT format); VEA is not fully annotation-free at the model-adaptation stage.
• LLaVA-Next and InternVL3.5 exhibit memory leakage during attention extraction, requiring Qwen2.5VL as a proxy model for attention-based image augmentation — limiting direct per-model adaptation.
• Evaluation is restricted to evidence retrieval-style VQA (text in scenes/documents); generalization to open-ended or abstract visual reasoning tasks is not demonstrated.
• The attention sink filtering heuristic (\(\lambda=10\)) and smoothing strength \(\sigma\) are hand-tuned hyperparameters.
• The analysis assumes a decoder-only transformer architecture; applicability to encoder-decoder VLMs is not addressed.

Relevance to Vision-Language Models

This paper directly targets a fundamental reliability question for VLMs: whether errors stem from perceptual or reasoning/generation failures. The finding that deep transformer layers reliably encode correct visual evidence even during wrong answers reframes the problem — it is not a vision encoder deficiency but a cross-modal integration failure within the LLM backbone. VEA's training-free nature makes it immediately applicable to any deployed VLM, complementing PEFT-based and RLHF-based visual grounding approaches. The attention-sink denoising strategy is also broadly relevant to any method that interprets VLM attention maps as grounding signals.