Skip to content

MRAG-Bench: Vision-Centric Evaluation for Retrieval-Augmented Multimodal Models

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MRAG-Bench is a vision-centric benchmark for retrieval-augmented generation (RAG) in LVLMs, covering 9 scenarios where retrieving images outperforms retrieving text. Evaluating 14 models reveals that even the best proprietary model (GPT-4o) gains only 5.82% from ground-truth visual retrieval, versus 33.16% for humans — exposing a critical gap in LVLMs' ability to exploit retrieved visual knowledge.

Problem

Existing multimodal RAG benchmarks (OK-VQA, InfoSeek, WebQA) are text-centric: their questions are solvable with retrieved textual knowledge. They do not evaluate the distinct and practically important case where visual retrieval is either easier or more informative than text retrieval — e.g., recognizing a car from an unusual angle, or tracking biological changes in an organism.

Method

The authors construct MRAG-Bench by identifying and categorizing 9 real-world scenarios where visual augmentation is strictly more useful than text. Scenarios split into two aspects: - Perspective (angle, partial, scope, occlusion): the query image shows a visual entity from a non-canonical viewpoint; retrieved images from canonical viewpoints resolve ambiguity. - Transformative (temporal, deformation, incomplete, biological): the entity has undergone physical change; retrieved images of the intact/standard state provide necessary reference.

Images are sourced from ImageNet, Oxford Flowers102, StanfordCars, Bing Image Search, and GeoDE; all 1,353 multiple-choice questions are manually annotated. Ground-truth image corpora (9,673 human-selected images) are included. Retrieval uses CLIP by default; analysis also covers MagicLens, E5-V, and VISTA. Evaluation tests models under three conditions: no RAG, retrieved-image RAG (top-5), and ground-truth image RAG (top-5).

Key Contributions

  • First benchmark explicitly designed for vision-centric multimodal RAG, with systematic scenario taxonomy (16,130 images, 1,353 human-annotated MCQs across 9 scenarios).
  • Empirical confirmation that all 14 tested LVLMs benefit more from visual augmentation than from equivalent textual augmentation.
  • Finding that open-source LVLMs degrade under noisy retrieved images (inability to filter bad examples), while proprietary models maintain or improve performance.
  • Quantitative analysis of retriever quality impact (95% correlation between Recall@5 and end-model accuracy) and optimal number of GT examples (~10 images).
  • Human performance baselines showing a large human–model gap in leveraging retrieved visual knowledge (+33.16% humans vs. +5.82% GPT-4o with GT images).

Results

  • GPT-4o (best overall): 68.68% without RAG → 74.50% with GT image RAG (+5.82%); humans: 61.38% → 94.54% (+33.16%).
  • LLaVA-OneVision (best open-source): 53.29% without RAG → 58.98% with GT image RAG (+5.69%).
  • All open-source models degrade with noisy retrieved images (e.g., mPLUG-Owl3: −7.91% overall); proprietary models improve or stay flat.
  • Visual GT RAG outperforms text GT RAG: LLaVA-Next-Interleave gains +11.90% from image vs. text GT; GPT-4-Turbo gains +3.87%.
  • LLaVA-Next-Interleave accuracy correlates 95% with retriever Recall@5 across four retrievers.
  • Performance peaks at 10 GT examples for LLaVA-Next-Interleave; marginal drop with 20 examples.
  • GPT-4o surpasses human performance on all perspective scenarios and temporal/deformation, but lags on incomplete and biological scenarios.

Limitations

  • Benchmark covers only multiple-choice format; open-ended generation RAG is not assessed.
  • Ground-truth corpus size varies across scenarios (average 5.9–20.4 images per question), which may favor scenarios with more images.
  • CLIP retriever used as default may not reflect state-of-the-art retrieval pipelines in real systems.
  • Position bias among retrieved images (order sensitivity) is noted as unexplored in the visual RAG setting.
  • The "Others" geographic scenario uses only 3 of 6 regions per question, limiting coverage diversity.
  • Human evaluation protocol differences from model evaluation may confound the human–model gap analysis.

Relevance to Vision-Language Models

MRAG-Bench directly probes a known weakness of current LVLMs: integrating multi-image context from retrieved sources rather than relying solely on parametric knowledge. The benchmark exposes that even frontier proprietary models like GPT-4o are far below human-level utilization of retrieved visual information — a gap that is orthogonal to raw visual understanding and points toward a specific failure mode in cross-image reasoning and evidence weighting. For researchers working on multi-image LVLMs, visual RAG, or in-context visual learning, MRAG-Bench provides the first principled evaluation suite with scenario-level granularity to diagnose where retrieval utility breaks down.