How Do VLMs Fail? Vision-Operation Misalignment in Compositional VQA¶

🕒 Published (v1): 2026-07-17 16:25 UTC · Source: Arxiv · link

Why this paper was selected

Vision-operation misalignment taxonomy; actionable failure map for compositional VQA

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper introduces an operation-centric mechanistic framework to explain why VLMs fail on compositional VQA, decomposing failures by both the reasoning operation where they originate and the internal computational pathway through which they propagate. Four mechanistically distinct failure modes are identified—grounding, reasoning, attribute extraction, and language prior dominance—each characterized by a unique relationship between visual grounding strength and answer correctness. A key finding is a pathway dissociation: grounding/attribute-extraction failures route through feedforward (MLP) networks, while reasoning failures route through late-layer attention.

Problem¶

Aggregate accuracy metrics on compositional VQA benchmarks (e.g., GQA) mask systematic failure patterns; prior mechanistic interpretability work focuses on broad failure types (object hallucination, cross-modal flow) and does not distinguish whether different reasoning operations—select, relate, verify, query, exist, filter, choose—fail through distinct internal mechanisms. There is no framework that maps failure type to the specific computational pathway (attention vs. feedforward) responsible.

Method¶

The study analyzes Qwen2.5-VL-3B-Instruct (36-layer transformer, native-resolution vision encoder) on 3,500 GQA questions (500 per operation type). The framework has three components:

Operation-aware scene-graph grounding. GQA functional program annotations and scene graphs are used to identify the exact vision token subset $\mathcal{B}_t$ causally relevant to each reasoning step, rather than ablating all visual tokens uniformly. Grounding sets are defined by causal structure (e.g., for relate, only the terminal answer-determining object is included).
Causal mean ablation. Object tokens $\mathcal{B}_t$ at layer $\ell$ are replaced with the mean activation over all vision tokens: $$\mu^{(\ell)} = \frac{1}{|\mathcal{V}|} \sum_{j \in \mathcal{V}} H_j^{(\ell)}$$ Degradation in log-probability of the correct answer $\Delta_t^{(\ell)} = \log p(a|I,q) - \log p(a|I,q;\text{ablate}(\mathcal{B}_t,\ell))$ characterizes causal reliance; peak degradation defines grounding strength $\text{GS}(o_t) = \max_\ell \Delta_t^{(\ell)}$.
Targeted pathway knockouts. Two complementary interventions isolate each pathway independently:
Attention knockout: zero attention weights from the answer position to $\mathcal{B}_t$, renormalizing the remainder—blocks direct attention routing without touching MLP.
MLP knockout: zero MLP outputs at either object positions ($\mathcal{B}_t$) or the answer position, blocking within-token feedforward transformation.

Failure modes are classified by the grounding specificity shift (Cohen's $d$) across all three degradation metrics, stratified by model correctness.

Key Contributions¶

Operation-aware failure taxonomy: Four mechanistically distinct failure modes defined by the joint signature of three causal interventions, covering all seven GQA operation types.
Scene-graph-grounded operation targeting: Interventions target causally relevant vision tokens per operation, not all visual tokens uniformly; grounding strength predicts correctness positively for select, inversely for relate, and not at all for language-prior operations.
Pathway dissociation: Empirically demonstrated that grounding and attribute-extraction failures are MLP-mediated, while reasoning (spatial) failures are attention-mediated at late layers—validated on the VSR spatial benchmark and cross-architecture on LLaVA-1.5-7B.

Results¶

Grounding failure (select, acc=60%): Correct answers show substantially higher grounding strength than incorrect ones ($d_{\text{abl}} > 0$); MLP knockout at object positions is the primary degradation pathway.
Reasoning failure (relate): Incorrect answers show higher grounding strength than correct ones (over-grounding, $d_{\text{abl}} < 0$, $d_{\text{attn}} < 0$); late-layer attention knockout ($d_{\text{attn}} = -0.31$ for verify-class) is the discriminating signal.
Attribute extraction failure (verify, acc=86%): Grounding strength is high regardless of correctness ($d_{\text{abl}} \approx +0.10$); answer-position MLP knockout is the discriminating pathway ($d_{\text{mlp}} = -0.80$).
Language prior dominance (filter, acc=90%): Grounding strength is low for both correct and incorrect ($d_{\text{abl}} = -0.08$, $d_{\text{attn}} = +0.07$, $d_{\text{mlp}} = -0.06$); no pathway carries correctness-discriminating visual signal.
Counterfactual probe validation: Vision-sensitive vs. vision-insensitive gap $\delta_{\text{VS}} > 0.05$ for all vision-dependent failure modes (grounding, reasoning, attribute extraction); filter fails this threshold ($\delta_{\text{VS}} = +0.03$), confirming its language-prior classification.
Cross-architecture: Attribute extraction and language-prior modes are architecture-independent; grounding and reasoning modes are encoder-dependent (confirmed on LLaVA-1.5-7B).

Limitations¶

Analysis is restricted to a single primary model (Qwen2.5-VL-3B-Instruct); cross-architecture validation uses only LLaVA-1.5-7B, leaving larger or proprietary models unexamined.
Only direct (single-layer) attention paths from the answer token to $\mathcal{B}_t$ are blocked; indirect multi-hop paths through intermediate tokens are not ablated, limiting completeness of pathway attribution.
GQA's chained operations make it difficult to isolate spatial reasoning from upstream grounding errors; VSR generalization partially addresses this for relate only.
The framework does not yet prescribe or evaluate concrete corrective interventions; it establishes diagnostic dissociation but not remediation.
choose operation is excluded from representational validation due to experimental constraints.

Relevance to Vision-Language Models¶

This work directly advances mechanistic interpretability for VLMs by moving beyond behavioral evaluation to causal pathway localization within the transformer, a crucial gap as VLMs scale and compositional reasoning failures remain poorly understood. The pathway dissociation finding—that feedforward networks mediate perceptual failures while attention mechanisms mediate relational reasoning failures—provides a principled basis for targeted architectural interventions rather than uniform fine-tuning. For researchers tracking VLMs, the operation-aware grounding methodology is immediately applicable to other compositional benchmarks and other model families, offering a reusable diagnostic toolkit. This connects to the broader line of work on mechanistic interpretability (attention circuits, MLP key-value memories) and bridges it with VLM-specific concerns like cross-modal grounding, hallucination, and attribute binding.