Ask Twice, Look Twice: Prompt Echoing Resolves the Question-First Paradox in Vision-Language Models¶

🕒 Published (v1): 2026-07-17 02:16 UTC · Source: Arxiv · link

Why this paper was selected

Deva Ramanan (CMU); prompt-position paradox fix; universal VLM prompting impact

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Placing the question before the image in VLM prompts ("question-first") consistently underperforms the image-first order, a paradox the authors trace to a dissociation between two mechanisms: question-first steers visual encoding but is under-attended at answer generation. A training-free fix—"question echoing" (restating the question on both sides of the image)—resolves the paradox and outperforms either single-ordering baseline.

Problem¶

Modern VLMs flatten system, image, and text tokens into a single causal sequence, making prompt ordering a silent design choice. Intuition suggests placing the question before the image (STI) should direct visual attention, yet empirically STI underperforms image-first ordering (SIT) by up to 17.5 group-accuracy points across benchmarks. No prior work mechanistically explains why question-first hurts or derives a principled fix.

Method¶

The authors probe the paradox with two mechanistic tools applied to NaturalBench "disagreement pairs" (150 pairs where SIT is correct and STI is wrong) on Qwen3-VL-8B:

Perception probe (logit lens): Projects each image patch's hidden state through the output embedding layer-by-layer to measure whether patches shift toward question-relevant vocabulary under STI vs. SIT. Cosine similarity relative to an image-only baseline quantifies the shift.
Read-out probe: Tracks the answer token's layer-wise attention over the question span and image span, plus the logit-lens probability of the correct answer token by depth.
Causal attention knockout: Adds a pre-softmax \(-\infty\) bias to all attention edges from the answer position to a target span (question, image, or random control) across all 36 text layers in a single forward pass, directly testing whether each edge is causally necessary.

The diagnosis yields two training-free prompt variants: - STIT (question echoing): System, Task, Image, Task — the pre-image copy steers perception; the post-image copy restores question–answer adjacency. - SITIT (image echoing): System, Image, Task, Image, Task — the second image pass encodes question-specific features and, under the causal mask, every patch of the second copy attends over all patches of the first, recovering a bidirectional whole-image read without architecture changes.

Key Contributions¶

Establishes the question-first paradox: STI trails SIT by 8.1 NaturalBench and 9.5 Winoground group-accuracy points on the primary model, reproduced on four of five VLMs.
Mechanistic diagnosis: logit-lens probes confirm STI genuinely steers patches toward question-relevant concepts (mean cosine shift 0.86 vs. 0.91 for SIT), yet the answer under-attends the question (peak attention 0.068 vs. 0.148), committing to a wrong image-anchored token in 41% of disagreement pairs vs. 2% for SIT.
Causal confirmation: attention knockout shows the answer causally reads the question under SIT (\(\Delta = -0.056\), \(p \approx 0.003\)) but not under STI, where the image edge drives the answer (\(\Delta = -0.096\), \(p \approx 0.001\)).
Training-free fix: STIT and SITIT, pure prompt edits that close or reverse the paradox gap across all tested models and benchmarks.

Results¶

NaturalBench (Qwen3-VL-8B): STI 0.270 → SIT 0.351 → SITIT best (≈+0.024 over STIT, +0.104 over STI).
Winoground group accuracy (Qwen3-VL-8B): STI 0.223 → SITIT 0.410, a +19 point gain over question-first (\(p < 10^{-13}\)); SITIT beats best SIT baseline by +9.2 points (\(p < 10^{-5}\)).
Winoground (Gemma-3-27B): climbs from 0.213 (STI) to 0.318 (SITIT), +11.0 points over SIT baseline (\(p < 10^{-6}\)).
LLaVA-1.5-7B (extreme case): STI collapses to constant "Yes" on every benchmark (NaturalBench group 0.000, POPE 0.500); SIT repairs it (POPE 0.871, \(\Delta = +0.371\)).
Qwen2.5-VL-7B: paradox gap reaches +0.175 NaturalBench, +0.210 Winoground; SITIT recovers where STIT under-recovers.
POPE: near-saturated across orderings except STI; differences small except for LLaVA.
Ablations rule out "more image tokens" and "more pre-answer compute" as confounds, isolating question adjacency and in-distribution image-first pairing as active ingredients.

Limitations¶

Results are confined to five open VLMs; closed/proprietary frontier models are not tested.
SITIT roughly doubles image token count, increasing inference cost proportionally; the paradox gap already plateaus with image tokens in the STI–SIT comparison, so costs may not always be justified.
The paradox is muted on Gemma-3-27B (which uses a bidirectional attention pattern over image blocks), suggesting the effect is architecture-dependent; it is largest on weaker or purely causal decoders.
Causal knockouts sever direct attention edges but not indirect information flow through intermediate tokens, so the measured causal effects are lower bounds.
Open-ended VQAv2 evaluation is mentioned but only partially reported in the provided text; claims of gains on that benchmark are incompletely detailed.

Relevance to Vision-Language Models¶

This paper directly addresses how the causal-decoder architecture of instruction-tuned VLMs interacts with multimodal prompt structure, a foundational design question for any VLM application. The mechanistic decomposition—showing that visual steering and decoder read-out are dissociable, positionally dependent mechanisms—provides an interpretability foothold that generalizes beyond prompt engineering to understanding how VLMs process interleaved token streams. The training-free echoing fix is immediately deployable and directly improves compositional VQA performance, which is a longstanding weakness of VLMs. The connection to the "lost-in-the-middle" phenomenon and the parallel to human adjunct-question research situates the finding within a broader theory of sequential comprehension in both artificial and biological systems.