Skip to content

Does It Fail to See or Fail to Know? Attributing Errors in Vision-Language Models

🕒 Published (v1): 2026-07-06 05:11 UTC · Source: Arxiv · link

Why this paper was selected

Disentangles perception vs. knowledge failures in VLM errors — useful for evaluation design

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLMs failing at knowledge-intensive VQA can fail for structurally distinct reasons — insufficient visual evidence, unknown entity, missing parametric fact, or failed cross-modal recall — and these failure sources can be predicted from pre-generation internal representations before the model decodes any tokens. Visual-token features dominate for recognition-side failures, while prompt-boundary hidden states are most informative for factual failures.

Problem

Knowledge-intensive VQA requires a multi-step chain (visual perception → entity recognition → fact retrieval → generation), yet prior work either studies individual failure modes in isolation or treats all incorrect answers as a single undifferentiated event. This prevents routing erroneous predictions to the appropriate targeted intervention (image repair, entity recognition support, or retrieval augmentation) and conflates structurally different failure causes under a single reliability score.

Method

Attribution-tree taxonomy. Four mutually exclusive failure components are defined: \(Z = (Z_d, Z_u, Z_f, Z_r)\), where \(Z_d\) = insufficient visual evidence, \(Z_u\) = unknown/unrecognized entity, \(Z_f\) = unknown fact absent from parametric memory, \(Z_r\) = failed factual recall (entity recognized, fact presumably available but not retrieved). Binary error probability decomposes as: $\(P(E{=}1 \mid x) = P(Z_d{=}1 \mid x) + P(Z_u{=}1 \mid x) + P(Z_f{=}1 \mid x) + P(Z_r{=}1 \mid x).\)$

Data curation. Seeds are drawn from PopVQA (4,863 pairs) and iNaturalist-EncyclopedicVQA (2,116 pairs) after filtering binary/disjunctive questions and non-visually-grounded items via a white-canvas ablation test. Recognition failures use yes/no probes with three entity distractors; visual-evidence failures are induced by progressive image corruption (Gaussian blur + noise + JPEG + downsampling) until recognition flips. Unrecallable-fact vs. unknown-fact is determined by replacing the image-grounded question with an entity-explicit text version (e.g., "Where is Tom Cruise from?" instead of "Where is this actor from?"): failure on both → unknown fact; failure only on visual version → unrecallable fact.

Pre-generation probes. Five feature sets are extracted after processing the image-question prompt but before decoding: VIS (hidden state at the final image token), EOP (end-of-prompt token hidden state), LAST8 (last 8 prompt token hidden states), ATTN (flattened attention weights from last 8 tokens to a 4-token lookback), and ATTN+LAST8 (combined). All features concatenate mid-to-late layer offsets \((-1, -4, -8, -12)\). Linear or two-layer transformer probes are trained at each node of the attribution tree using PR-AUC as the metric (to handle class imbalance). Four VLMs are tested: Gemma-3-12B-it, Llama-3.2-11B, Qwen2.5-VL-7B, Qwen3-VL-8B.

Key Contributions

  • Attribution-tree taxonomy partitioning knowledge-intensive VQA failures into four mutually exclusive, operationally grounded components.
  • Scalable model-specific labeling procedure that assigns instances to attribution nodes using controlled recognition probes, image corruption, and entity-explicit question variants.
  • Empirical demonstration that VIS features dominate for recognition-related failures while LAST8/prompt hidden states dominate for post-recognition factual failures — a structured, pre-generation diagnostic signal.
  • Systematic comparison against post-generation baselines (OutSeq, TokProb) and UQ baselines (MSP, Semantic Entropy, SAR, ORACLE-UQ), showing pre-generation probes match or exceed them.

Results

  • Visual-evidence failure attribution (VIS, linear): average PR-AUC 97.0, vs. next best pre-gen feature ATTN+LAST8 at 67.6, and post-gen TokProb at 59.1.
  • Entity recognition failure (VIS, linear): average PR-AUC 86.7, vs. ORACLE-UQ post-gen baseline at 77.7.
  • Post-recognition answer success (LAST8, linear): average PR-AUC 62.7, vs. post-gen OutSeq (Tr.) at 56.2 and ORACLE-UQ at 31.4.
  • Unknown vs. unrecallable fact (ATTN+LAST8, linear): average PR-AUC 49.0 — near-random baseline of 49.8 — indicating this distinction is the hardest to predict pre-generation; MSP at 47.9.
  • Model success rates are low: best is 34.9% (Gemma on PopVQA) and 20.8% (Llama on iNaturalist). UNKNOWN FACT is the dominant failure (50.7–63.8% of iNaturalist errors, 36.8–37.7% of PopVQA errors).
  • Qwen3 is an exception: ATTN+LAST8 (87.1) beats VIS (81.4) on recognition, suggesting its recognition behavior is better reflected in integrated prompt states than in visual tokens.

Limitations

  • Visual-evidence failures are artificially induced (controlled corruption), not drawn from naturally degraded in-the-wild images; generalization to organic degradation is unverified.
  • Only four open-weight VLMs tested; closed-source or larger models are excluded.
  • Attribution tree assumes mutually exclusive failure modes, which is an operational approximation and may not hold when failures have compound causes.
  • The unknown-fact vs. unrecallable-fact distinction is not reliably predicted pre-generation (near-random PR-AUC), limiting the actionability of that branch.
  • Datasets are restricted to celebrity, landmark, logo, painting, and naturalist species domains; coverage of other knowledge-intensive VQA domains is unknown.

Relevance to Vision-Language Models

This work directly targets a fundamental diagnostic gap in VLM evaluation: rather than scoring correctness, it decomposes failures structurally to expose which modality or knowledge stage broke down. The finding that pre-generation visual-token vs. prompt hidden-state representations encode complementary diagnostic signals contributes mechanistic understanding of how VLMs route visual and factual information in their decoder. For the VLM research community, the framework provides a principled basis for conditional intervention (RAG only when parametric knowledge is absent, image repair only when visual evidence is degraded), moving uncertainty quantification from abstention toward actionable routing.