Skip to content

Lost at the End: Primacy Bias in Multimodal Retrieval-Augmented Question Answering

🕒 Published (v1): 2026-06-15 09:57 UTC · Source: Arxiv · Venue: EMNLP 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

In multimodal knowledge-based VQA (KB-VQA), the "lost-in-the-middle" U-shaped position bias observed in text LLMs does not transfer: instead, VLM readers exhibit a strong primacy bias, where gold passages at prompt slot 0 outperform gold at the last slot by 16–26 percentage points across all tested reader–benchmark combinations. The multimodal setting amplifies an underlying text-mode primacy effect by 2.2–4.5×, and the locus is prompt slot 0 of the instruction-tuned reader, not image-token proximity or retrieval similarity. Retrieval-side fixes (MMR, oracle reranking, distractor reordering) cannot close the gap on a frozen reader, implicating reader-side intervention as the necessary remedy.

Problem

Prior work on position bias in long-context LLMs established a U-shaped "lost-in-the-middle" effect, but it was unknown whether this shape holds for deployed multimodal KB-VQA systems operating at Wikipedia-scale with strong retrievers and up to 20 competing passages. Existing multimodal-RAG position-bias studies used toy-scale distractor counts (2–19) and general-image benchmarks, not KB-VQA stacks, and lacked controlled within-question designs that isolate the reader's position response from retrieval rank, distractor identity, and scoring confounds.

Method

The authors design a gold-position protocol: for each question, three k-passage prompts are constructed that are bit-identical except for the index of the gold (relevant) passage—placed at slot 0 (first), slot ⌊(k−1)/2⌋ (middle), and slot k−1 (last). Nine distractors are held fixed in identity and order across all three cells per question, enabling exact paired-bootstrap inference on between-cell accuracy deltas. The probe runs on:

  • Readers: Qwen2.5-VL-7B, InternVL3-8B, Qwen3-VL-8B (frozen, greedy, instruction-tuned)
  • Benchmarks: InfoSeek (98K-passage KB) and E-VQA (51K-passage KB), n≈496–500 per cell
  • Retriever: PreFLMR ViT-G top-50 candidates; k swept over {1, 3, 5, 10, 20}

Three targeted ablations narrow the mechanism: 1. Text-only ablation: same readers on NQ-Open with image block omitted, to quantify multimodal amplification. 2. Image-at-end ablation: image tokens moved after the passage block to test image-token proximity as a driver. 3. Distractor-shuffle ablation: gold fixed at last slot; distractors placed in rank-order (A), reverse-rank (B), or random (C) order—dissociating slot-0 anchoring from retrieval-similarity anchoring.

Three retrieval-side fixes on a frozen reader are also evaluated: MMR diversification (λ ∈ {0.3, 0.5, 0.7}, k ∈ {5, 10}), oracle reranking (gold forced to first/last), and rank-based distractor reordering.

Key Contributions

  • First controlled, within-question probe of reader-side position bias in deployed multimodal KB-VQA at deployment scale (top-50 distractor pool, k up to 20, three reader families, two benchmarks).
  • Establishes that the shape is primacy, not U: gold-at-first beats gold-at-last by 16–26 pp on all six reader×benchmark cells, with middle and last collapsing together.
  • Quantifies multimodal amplification: the multimodal setting amplifies text-mode primacy by 2.23–4.47× across three reader families.
  • Locus identification: image-token proximity and retrieval similarity are ruled out as primary drivers; the effect anchors specifically to prompt slot 0 of the instruction-tuned reader.
  • Retrieval-side null result: MMR, oracle reranking, and distractor reordering all fail to close the gap on a frozen reader within paired-bootstrap CIs.
  • Releases the probing protocol as a reusable instrument for evaluating reader-side interventions.

Results

  • Primary effect (k=10): Qwen2.5-VL-7B × InfoSeek — 45.2% (first) vs. 28.6% (last), Δ = +16.5 pp; Qwen2.5-VL-7B × E-VQA — 63.2% vs. 36.8%, Δ = +26.4 pp.
  • All six reader×benchmark first-vs-last deltas range +16.1 to +26.4 pp and all classify as GENERALIZATION under the pre-registered classifier.
  • Text-only ablation: first-vs-last gaps of +7.4 pp (Qwen2.5-VL-7B), +5.2 pp (InternVL3-8B), +6.4 pp (Qwen3-VL-8B) on NQ-Open; multimodal raises these to 16.1–26.4 pp (2.23–4.47× amplification).
  • Image-at-end ablation: first-vs-last gap remains +18.55 pp (1.12× anchor), rejecting proximity as primary driver.
  • Distractor-shuffle: in dissociable cells B+C, slot-0 distractor selection runs at 3.83–4.42× chance baseline; rank-1 distractor selection sits at 1.00–1.01× chance—reader follows slot, not retrieval rank.
  • MMR diversification: all paired-bootstrap CIs on Δ(MMR − vanilla) overlap zero; maximum point gain +1.00 pp.
  • Oracle reranking (n=254 gold-in-top-10 subset): first-vs-native Δ = +3.15 pp (not separable from zero); last-vs-native Δ = −5.51 pp (separable negative).
  • Exception: Qwen3-VL-8B × E-VQA shows a separable last > middle effect at k=10 (+3.0 pp) and k=20 (+5.2 pp)—the only recency rebound across all six panels.

Limitations

  • Empirical scope confined to three open-source 7B/8B instruction-tuned VLMs; generalization to proprietary models (GPT-4V) or larger scales is untested.
  • Text-only ablation confounds modality and corpus (NQ-Open vs. M2KR); an M2KR-text control is absent.
  • Mechanism at slot 0 is unidentified: instruction-tuning data distribution, positional encoding (RoPE), and attention concentration are candidate accounts but not directly tested.
  • Oracle reranking cell is underpowered (n=254); first-vs-native directional gain does not clear the noise floor.
  • Failure-bucket diagnostic is string-match heuristic, not causal; subject to paraphrase blind spots.
  • Single image per question, greedy decoding, English-only, k ≤ 20.
  • Retrieval-side mitigations tested are training-free only; learned rerankers and self-reflective selectors are out of scope.

Relevance to Vision-Language Models

This paper directly characterizes a systematic failure mode of instruction-tuned VLMs in RAG pipelines: their strong slot-0 anchoring means that standard recall@k retrieval metrics are decoupled from deployed accuracy, a result with immediate engineering implications for any KB-VQA or multimodal RAG system built on frozen VLM readers. The 2.2–4.5× multimodal amplification over text-mode primacy suggests that image conditioning interacts with positional encoding or attention patterns in ways not yet understood, pointing to a gap in alignment between instruction-tuning data distributions and multi-passage retrieval contexts. For VLM researchers, the findings motivate reader-side interventions—position-diverse fine-tuning data, attention calibration, and permutation-aware training—as higher-leverage investments than retrieval-side engineering. The controlled within-question probe protocol itself is a methodological contribution applicable to evaluating future VLMs for position robustness.