Same Task, Different Circuits: Disentangling Modality-Specific Mechanisms in VLMs¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

VLMs consistently underperform on visual tasks relative to analogous textual tasks. This paper uses mechanistic circuit analysis to show that vision and language circuits are structurally disjoint yet functionally similar in query/generation processing, with the gap originating in data-token processing. A training-free "back-patching" intervention—re-injecting later-layer visual token representations into earlier layers—closes ~32% of the modality performance gap.

Problem¶

Despite impressive VLM capabilities, a persistent accuracy gap exists between visual tasks (e.g., counting objects in an image) and structurally analogous textual tasks (e.g., counting words in a list). The mechanistic source of this gap—whether structural, functional, or representational—was unknown.

Method¶

The authors build a dataset of five task pairs (object counting, two-operand arithmetic, spatial ordering via CLEVR, factual recall, sentiment analysis), each with an image variant and an analogous text variant. They apply attribution patching with integrated gradients (AP-IG) to score component importance and extract minimal circuits (attention heads and MLP neurons at specific output positions) achieving >80% faithfulness on three VLMs (Qwen2-7B-VL-Instruct, Pixtral-12B, Gemma-3-12B-it).

Circuits are partitioned into data, query, and generation sub-circuits by token position. Structural overlap is measured via bidirectional normalized IoU; functional overlap via interchange faithfulness (swapping sub-circuits between modalities and measuring downstream faithfulness drop).

To exploit the finding that visual tokens gradually align to textual representations in later layers, they apply back-patching: a first forward pass records activations at a source window of layers (from the later half of the network); a second forward pass replaces the activations at an earlier destination window with the recorded ones, specifically for visual token positions. Optimal source/destination layers are selected by grid search.

Key Contributions¶

Demonstration that vision and language circuits for analogous tasks are largely structurally disjoint (only ~18% component overlap on average).
Evidence that query and generation sub-circuits are functionally interchangeable across modalities, while data sub-circuits are functionally incompatible (interchange faithfulness drops to random-baseline level).
Cosine-similarity analysis showing visual token activations align with textual counterparts only in late layers, after information has already propagated to downstream positions.
A training-free back-patching method that injects late-layer visual representations into early layers, yielding +4.6% average absolute accuracy on visual tasks and closing ~32% of the vision–text performance gap across five tasks and three VLMs.

Results¶

Average structural circuit overlap between modalities: ~18% shared components (data sub-circuit NIoU ≈ 0, query ≈ 12%, generation ≈ 38%).
Interchange faithfulness: query and generation sub-circuits maintain high faithfulness when swapped; data sub-circuit interchange degrades to random baseline across all tasks/models.
Back-patching absolute gains (selected highlights):
Qwen2-7B-VL: Arithmetic +9.4%, Counting +4.2%, Sentiment +2.2%
Pixtral-12B: Sentiment +17.7%, Arithmetic +6.7%, Color Ordering +4.1%
Gemma-3-12B: Arithmetic +6.7%, Sentiment +5.4%, Counting +2.0%
Average absolute improvement: +4.6%, closing ~32% of the modality gap.
Control experiment on textual prompts shows back-patching yields smaller gains in ~79% of cases, confirming the effect is specific to visual alignment rather than extra compute.

Limitations¶

Analysis restricted to single-image, single-token-completion QA tasks; multi-image, multi-token, or complex visual reasoning tasks not covered.
Current circuit discovery methods only support single-token completions, limiting scope by methodology, not just choice.
Only adapter-based VLMs studied; findings may not transfer to other vision-language fusion architectures.
Back-patching closes only ~32% of the gap; causes of the remaining gap are unidentified.
Circuit extraction requires prompt templates with fixed positional alignment, filtering out prompts that don't conform.

Relevance to Vision-Language Models¶

This paper provides a rare mechanistic account of why VLMs underperform on visual versus textual tasks, attributing the gap to data-token processing circuits rather than query comprehension or answer generation. The finding that query and generation sub-circuits are functionally equivalent across modalities suggests that the language backbone generalizes well to multi-modal prompts everywhere except at the visual input interface. The back-patching intervention is directly actionable as an inference-time augmentation for any adapter-based VLM, with no fine-tuning required. For researchers studying VLM alignment, hallucination, and representational gaps, this work bridges mechanistic interpretability and practical performance improvement in a unified framework.