Skip to content

MuirBench: A Comprehensive Benchmark for Robust Multi-image Understanding

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MuirBench is a comprehensive benchmark for evaluating multi-image understanding in multimodal LLMs, comprising 11,264 images and 2,600 multiple-choice questions across 12 task types and 10 multi-image relation categories. A key design feature is pairwise construction: every answerable question is paired with an expert-crafted unanswerable variant with minimal semantic difference to stress-test robustness. Even GPT-4o achieves only 68.0% accuracy versus 93.2% human performance, revealing large gaps in holistic multi-image reasoning.

Problem

Existing VLM benchmarks overwhelmingly focus on single-image understanding, leaving multi-image reasoning capabilities largely uncharted. Prior multi-image benchmarks (e.g., MANTIS-Eval with 207 examples, BLINK) cover limited task types and relation categories, and none systematically evaluate whether models can recognize unanswerable questions—a critical real-world requirement. Models trained on single-image data do not generalize to multi-image settings, and no controlled evaluation framework existed to diagnose this gap.

Method

MuirBench is constructed as a multiple-choice QA benchmark with a pairwise answerable/unanswerable design. Data sourcing uses three strategies: reuse of existing datasets (40.8%, from GeneCIS, SeedBench, IconQA, NLVR2, HallusionBench, etc.), format-derived data reformatted into MCQA (21.7%), and newly collected data for underrepresented tasks such as geographic and visual retrieval (37.5%). Unanswerable counterparts are created via three manipulation strategies applied with minimal surface change: image replacement/reordering (24.2%), question modification (35.3%), and option replacement (40.5%). Fine-grained metadata labels (image position, image type, relation category) enable diagnostic sub-analyses. Evaluation uses VLMEvalKit with temperature 0; single-image models receive concatenated images as a fallback.

Key Contributions

  • A benchmark with 12 multi-image tasks (action understanding, attribute similarity, cartoon understanding, counting, diagram understanding, difference spotting, geographic understanding, image-text matching, ordering, scene understanding, visual grounding, visual retrieval) and 10 relation categories.
  • Pairwise answerable/unanswerable instance design that tests whether models know what they do not know, improving evaluation reliability against vision/language shortcuts.
  • Comprehensive evaluation of 20 multimodal LLMs (both proprietary and open-source, single- and multi-image-trained), establishing a detailed performance baseline.
  • Fine-grained diagnostic labels (image input position, image type, relation category, unanswerable type) enabling targeted error analysis.

Results

  • GPT-4o: 68.0% overall; human accuracy: 93.2% (gap of ~25%).
  • Gemini Pro: 49.3% overall (gap of ~44% vs. humans).
  • Open-source multi-image-trained models (Mantis-8B variants, VILA1.5-13B, Idefics2-8B): 23.73%–44.50%, often near or below random chance (25%).
  • Single-image-trained models (LLaVA variants, CogVLM, MiniGPT-4-v2): cluster below 33.3%, the random-choice baseline.
  • Answerable vs. unanswerable gap: GPT-4o drops 26.8 pp on unanswerable instances; Gemini Pro drops 21.5 pp.
  • Relative task difficulty: ordering and visual grounding are hardest; image-text matching, visual retrieval, and diagram understanding are relatively easier.
  • Image position effect: accuracy is highest when images appear as options; highest error rate when images appear mid-question.
  • GPT-4o error analysis (100 sampled cases): 26% failure to capture image details; 20% counting/reasoning errors; 18% logical reasoning errors; 14% cross-image object identification; 12% sequence intent inference.

Limitations

  • All questions are in English only, introducing potential bias for multilingual evaluation settings.
  • Unanswerable instances are hand-crafted by a small team of four annotators, which may not exhaustively cover all real-world unanswerability patterns.
  • Benchmark is static; it does not evaluate continual or few-shot adaptation to new multi-image relation types.
  • Single-image model fallback (image concatenation) may not reflect true single-image model capability on multi-image inputs, as concatenation conflates spatial layout.
  • No evaluation of models beyond those available at submission time (e.g., newer instruction-tuned open-source models).

Relevance to Vision-Language Models

MuirBench directly targets a core capability gap in VLMs: the ability to integrate and reason across multiple images rather than treating each image in isolation. The pairwise unanswerable design adds a robustness dimension rarely assessed in prior VLM benchmarks, exposing a tendency to hallucinate plausible answers rather than abstain. Results showing that multi-image training data matters more than model scale are actionable guidance for VLM training recipe design. The benchmark also differentiates among relation types (temporal, multiview, narrative, etc.), enabling targeted dataset curation to address specific VLM weaknesses.