Skip to content

Vision-Default, Prior-Override: Causal Mechanisms of Perception-Knowledge Conflict in Vision-Language Models

🕒 Published (v1): 2026-06-26 17:16 UTC · Source: Arxiv · link

Why this paper was selected

Mechanistic causal account of perception-knowledge conflict; VLM reliability foundation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLMs resolve perception-knowledge conflict asymmetrically: visual grounding is the default pathway, while prior (world) knowledge override depends on a sparse, causally necessary set of attention heads (2.5–4.8%) concentrated in the network's second half. Ablating these heads flips prior-grounded predictions in 68–96% of conflict cases while leaving visually grounded predictions almost unchanged (0.8–7.5%). This sparse routing-and-writing circuit generalizes across three VLM families and multiple scales.

Problem

When visual evidence and memorized world knowledge conflict (e.g., a blue strawberry asked "what color is a strawberry usually?"), VLMs exhibit a poorly understood resolution mechanism. Prior work characterizes this behavior and localizes it to mid-to-late layers but provides only behavioral or correlational accounts—no component-level causal identification of which specific heads or sublayers are necessary and sufficient for each grounding mode.

Method

The authors use the Visual-Counterfact dataset (469 counterfactual recolored images with paired Visual/Prior prompts) and apply activation patching at three granularities—residual stream \(r^{(\ell)}\), per-head output \(z_h^{(\ell)}\), and MLP sublayer output \(m^{(\ell)}\)—across two swap directions: - P2V (Prior→Visual): patch prior-grounded activation into visual-grounded forward pass. - V2P (Visual→Prior): patch visual-grounded activation into prior-grounded forward pass.

Restoration scores \(R_d(c)\) (logit-difference recovery) and discrete flip rates \(F_d(c)\) (fraction of examples where patching changes the argmax) quantify each component's causal contribution. Heads are classified via PCA on the joint \((R_{P2V}, R_{V2P})\) vector: those beyond \(\pm 2\sigma\) on PC1 are labeled promoting or suppressing. Complementary zero-ablation studies (individual and group) test necessity. Mechanistic characterization uses attention-pattern analysis (image vs. text token attention delta) and logit-lens projection of head-output differences to distinguish routing heads (redirect attention flow) from writing heads (directly project answer tokens into residual stream). Models tested: Qwen-VL 2.5 (3B, 7B), LLaVA-NeXT 7B (Mistral backbone), PaliGemma (3B, 10B).

Key Contributions

  • First component-level causal account of perception-knowledge conflict in VLMs, identifying specific attention heads and MLP sublayers mediating the decision.
  • Establishes a strongly asymmetric causal structure: visual grounding is the default (emerges without active injection), while prior grounding requires a sparse 2.5–4.8% of attention heads in the network's second half.
  • Decomposes classified heads into two functional roles: routing heads (early; modulate information flow) and writing heads (late; directly project answer tokens via logit-lens into vocabulary space).
  • Shows the circuit generalizes across three architecturally distinct VLM families, but routing implementation diverges: Qwen-VL/LLaVA-NeXT redistribute attention between image and text tokens (\(\Delta_{\text{img-attn}} \approx +0.19\)–\(+0.24\)), while PaliGemma routes through differences in attended representations with near-zero attention deltas.

Results

  • Behavioral baseline: Under no-conflict conditions, all five models achieve 86–96% accuracy. Under the conflict condition (Prior grounding, counterfactual image), accuracy collapses to 17.7–55.7%.
  • Critical window: Residual-stream patching localizes conflict resolution to a 7–16 layer window beginning at 52–76% of network depth. V2P restoration consistently rises before P2V across all models (lag of 3–19 layers to reach \(F_d = 0.5\)).
  • Sparse heads: Only 2.5–4.8% of all attention heads are classified as promoting/suppressing.
  • Group ablation flip rates (promoting attention heads):
  • Prior grounding flipped: Qwen-VL 3B 84.9%, Qwen-VL 7B 68.4%, LLaVA-NeXT 7B 75.0%, PaliGemma 3B 95.9%, PaliGemma 10B 88.7%.
  • Visual grounding flipped: 0.8–7.5% across all models.
  • MLP ablations: Directionally consistent but weaker (11–74% prior flips vs. 0–1.4% visual flips); effects diminish in larger models, supporting an amplifier rather than primary routing role.
  • Writing heads: A small subset of late-layer classified heads places the counterfactual color in the top-20 predicted tokens in >80% of examples via logit-lens projection.
  • Robustness check (image-varied contrast): Ablating promoting heads flips visual-grounded predictions in only 1.8–40.6%, reinforcing the asymmetry conclusion under an alternative experimental contrast.

Limitations

  • Analysis restricted to color-attribute conflict (Visual-Counterfact dataset); generalizability to other attribute types, relations, or knowledge domains is untested.
  • Patching uses the last token position only; the role of earlier token positions is not characterized.
  • The image-varied robustness check produces a wider range (1.8–40.6% for visual flips), suggesting partial overlap between the two contrasts that is not fully resolved.
  • Restricted to models in the 3B–10B parameter range; behavior at much larger scales is unknown.
  • The paper is cut off before full mechanistic characterization results for writing heads and before discussion/conclusion sections are presented.

Relevance to Vision-Language Models

This work provides the first mechanistic, causally validated account of how VLMs arbitrate between visual perception and parametric world knowledge—a core reliability question for deployed multimodal systems. By identifying the sparse attention-head circuit responsible for prior-knowledge override, it opens a direct path to targeted interventions (e.g., steering or fine-tuning a handful of heads) to improve calibration in settings where models should trust knowledge over percept, or vice versa. The routing-then-writing decomposition connects and extends prior correlational work on multimodal information routing and factual storage in MLPs to VLMs specifically. For VLM researchers, the cross-architecture generalization of the asymmetric circuit is a strong structural constraint that any mechanistic model of multimodal inference must respect.