Too Late to Recall: Explaining the Two-Hop Problem in Multimodal Knowledge Retrieval¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

VLMs systematically underperform their LLM backbones on factual recall because visual entity representations emerge too late in the forward pass, causing the model to bypass early-layer MLPs that implement factual recall. This "two-hop problem"—first resolving a visual entity, then retrieving associated facts—is diagnosed mechanistically via attribution patching, activation patching, and linear probing across 14 VLMs.

Problem¶

Adapter-based VLMs (LLaVA-style) show substantial factual recall degradation relative to their LLM backbones, yet the mechanistic cause was unknown. Prior work established that LLM factual recall depends on early-layer MLPs reading subject token representations, and separately that visual projections are not aligned with the LLM token space in early layers—but no work had connected these two observations to explain the failure mode.

Method¶

The authors frame factual recall from images as a two-hop problem: hop 1 = forming a grounded entity representation from visual tokens; hop 2 = accessing factual knowledge via the LLM's existing recall circuit. They construct a 15,000-question benchmark using WIT images and GPT-4.1-generated entity-specific factual questions, filtering out entity-misidentification errors to isolate pure recall failures. They then apply three mechanistic tools to five selected models (three high-degradation LLaVA, one native Gemma-3-12B, one massively fine-tuned Qwen2.5-VL-7B):

Attribution patching (Grad×Delta) over MLP and attention sublayers to identify which layers causally contribute to factual recall in both LLMs and VLMs.
Activation patching (heuristic): cache early-layer MLP outputs from the LLM backbone over entity tokens, then use a gradient-based position-attribution method to find the optimal injection point in the VLM (which has no entity token positions), and patch there; compare against random-position and back-patching baselines.
Linear probing of residual-stream representations layer-by-layer on ImageNet-100 entity classification to track when robust entity representations emerge.
Chain-of-thought prompting as an inference-time mitigation that converts visual entity recognition into text tokens before factual retrieval.

Key Contributions¶

Formal framing of multimodal factual recall as a two-hop problem and mechanistic explanation of why it fails in adapter-based VLMs.
Benchmark of 14 VLMs (7B–124B; LLaVA, Native, Cross-Attention architectures) against their LLM backbones, finding 11/14 exhibit significant degradation.
Attribution patching showing degraded VLMs lack the early-entity-token MLP site active in LLM backbones, while high-performing VLMs preserve it.
Causal confirmation via heuristic activation patching: injecting LLM early-layer MLP outputs recovers ~35% of the accuracy gap (vs. 13% back-patch, 16% random).
Linear probe analysis showing entity representations emerge at middle-to-late layers in LLaVA models but at all layers in Gemma-3 and Qwen2.5-VL.
Chain-of-thought as a practical mitigation; closes the gap fully for Pixtral-12B/Large and partially for LLaVA-1.5-13B.

Results¶

11 of 14 VLMs show factual recall degradation vs. their LLM backbone.
Largest drops (adapter-based): LLaVA-MORE-8B ↓43.9%, LLaVA-NEXT-8B ↓41.5%, LLaVA-1.5-7B ↓36.7%, LLaVA-1.5-13B ↓36.4%.
Pixtral-Large-124B (adapter) still drops 17.6% despite scale.
Near-zero degradation: Gemma-3-12B (0.0%), Gemma-3-27B (0.0%), GPT-4o (↓5.2%), Llama-4-Maverick (↓5.3%).
Qwen2.5-VL-72B slightly exceeds its backbone (+8.2%) after massive multimodal fine-tuning (4T tokens).
Heuristic patching recovers 35% of the performance gap on originally-wrong VLM examples vs. 13% (back-patch) and 16% (random).
CoT prompting: Pixtral-12B closes its full gap (+12pp VLM vs. +4pp LLM); Pixtral-Large-124B fully closes; LLaVA-1.5-13B closes >half; LLaVA-1.5-7B shows no improvement (−1pp).

Limitations¶

Patching experiments only cover three LLaVA-style degraded models; generalizability to other adapter architectures is not demonstrated.
Heuristic patching requires a separate attribution pass and is not a deployable fix—it is a causal probe, not a practical repair.
Chain-of-thought improvements are highly model-dependent; smaller/weaker models may not benefit or may regress.
Study is scoped to factual recall; other multimodal reasoning tasks may have different failure modes.
Native VLMs (GPT-4o, Gemini) cannot be probed mechanistically due to closed weights; analysis relies on behavioral benchmarking only for those models.
Benchmark discards entity-misidentification samples, so the method does not address the joint recognition+recall pipeline.

Relevance to Vision-Language Models¶

This paper provides one of the most precise mechanistic accounts of a known VLM failure—factual hallucination relative to the LLM backbone—by grounding it in the mismatch between visual representation alignment timelines and the fixed location of the factual recall circuit in LLMs. For researchers tracking VLMs, the key finding is that alignment is not just a representational problem but a temporal one within the forward pass: visual adapters must produce grounded entity tokens early enough to engage LLM circuits, not just eventually. The work directly implicates adapter design and multimodal fine-tuning data scale as the primary levers, and suggests chain-of-thought reasoning as an accessible inference-time mitigation. It also advances multimodal mechanistic interpretability methodology, demonstrating that attribution and activation patching techniques developed for LLMs transfer meaningfully to cross-modal settings.