Skip to content

OMIBench: Benchmarking Olympiad-Level Multi-Image Reasoning in Large Vision-Language Model

🕒 Published (v1): 2026-04-22 17:37 UTC · Source: Arxiv · Venue: ACL 2026 · link

Why this paper was selected

Olympiad-level multi-image reasoning benchmark for frontier VLMs; ACL 2026

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OMIBench is a competition-grade multimodal benchmark of 1,322 Olympiad-level problems across biology, chemistry, mathematics, and physics where the necessary evidence is distributed across multiple images (average 3.07 images/problem). It exposes a large unaddressed capability gap: the strongest LVLM (Gemini-3-Pro) reaches only ~50.5%, leaving a >30-point gap to human experts. Standard remedies—CoT prompting, test-time scaling, in-context learning, and think-with-image methods—yield only marginal improvement.

Problem

Existing Olympiad-level multimodal benchmarks are almost exclusively single-image; models that perform well on them have not been tested on problems requiring autonomous cross-image alignment, filtering, and integrative reasoning. General multi-image benchmarks (MMIU, Blink, MuirBench) cover basic perceptual tasks but lack competition-grade scientific difficulty. The intersection—multi-image + Olympiad-level—is unaddressed, and current LVLMs' performance there is unknown.

Method

Dataset construction: - 1,322 problems drawn from international/national Olympiads and high-stakes science competitions, covering 40+ sub-topics across four disciplines. - Raw PDFs processed with Mathpix OCR and normalized to a structured Question–Rationale–Answer format; multilingual items machine-translated then human-verified. - Rationale annotation is two-stage: Gemini-2.5-pro-thinking generates up to 16 candidate solutions per problem, candidates with correct answers are retained (if none, the ground-truth answer is provided and solutions are regenerated, saving ~20% human effort), then experienced annotators verify, correct, and standardize notation. - Quality control uses dual-review with weekly 5% random regression sampling and closed-loop feedback to annotation guidelines. - Classification labeling: GPT-4o generates preliminary topic labels; humans finalize.

Evaluation: - Two task types: multiple-choice (574 problems, 43.4%) and open-ended (748, 56.6%). - Exact-match accuracy for MC; GPTScore (semantic equivalence assessed by a judge LM under multimodal constraints) for OE; micro-averaged accuracy as the primary metric. - Models prompted with boxed-format (\boxed{·}) instructions; temperature selected from \(\{0, 1\}\).

Diagnostic experiments: cross-benchmark Spearman correlation, image-count ablations, information-equivalent single-image control (all images concatenated into one composite), human-expert baseline on a 52-problem subset, and four enhancement families: long CoT / thinking variants, three-axis test-time scaling (sequential token budget, parameter count, parallel sampling), multimodal ICL, and think-with-image (tool-based VisualSketchpad + internal generation via Emu-3.5).

Key Contributions

  • Identifies and formalizes the multi-image Olympiad-level reasoning gap as a distinct, underserved evaluation setting.
  • Releases OMIBench: 1,322 competition-grade problems, 100% rationale coverage, both MC and OE formats, four STEM disciplines, fine-grained sub-topic labels.
  • Establishes comprehensive baselines across open- and closed-source LVLMs, exposing a >30-point gap to human experts.
  • Provides diagnostic analyses isolating cross-image information flow (vs. visual volume or OCR noise) as the primary bottleneck, and benchmarks four enhancement directions with quantified results.

Results

  • Best model overall: Gemini-3-Pro-Preview, ~50.53% GPTScore on OMIBench.
  • Open-source vs. closed-source gap: Gemini-3-Pro-Preview outperforms the best open-source models by ~15%; GPT-4o scores only marginally above open-source models.
  • Single-image vs. multi-image drop: Gemini-3.0-Pro falls from 75.67% on OlympiadBench to 50.53% on OMIBench (>25 pp absolute); OMIBench produces performance drops >20% relative to existing multi-image benchmarks (MMIU).
  • Human baselines (52-problem subset): experts >80%; trained non-experts >57%; Gemini-3-Pro 48.08%—gap of >30 pp to experts and ~10 pp to non-experts.
  • Image-count sensitivity: single-image accuracy ~40%; drops below 15% for problems with ≥6 images. Concatenating all images into one composite raises accuracy by ≥10 pp, confirming multi-image organization as the bottleneck.
  • Spearman correlations: OMIBench vs. OlympiadBench \(\rho=0.614\); OMIBench vs. MMIU \(\rho=0.535\)—moderate, confirming distinct capability dimension.
  • Failure modes (GPTScore-annotated errors): logical reasoning fallacies 41%, cross-image association failures 29%, visual perception failures 13%.
  • Long CoT (thinking vs. instruct Qwen3-VL): thinking variant outperforms instruct by ~10 pp on OMIBench—notably larger than on MathVista (<5 pp). Standard CoT prompt variants show no statistically significant differences.
  • Sequential test-time scaling: increasing token budget from 512 to 16,384 yields near-monotonic gains.
  • Parameter scaling: InternVL and QwenVL plateau at ~25% and ~35% GPTScore respectively; performance scales positively against activated parameters (not total MoE parameters).
  • Parallel scaling (self-consistency, \(k\) samples, \(T=0.6\)): accuracy improves approximately log-linearly with \(k\).
  • MM-ICL: multi-image ICL > single-image ICL, but both underperform no-image ICL for cross-image reasoning.
  • Think-with-image (tool-based VisualSketchpad + internal Emu-3.5): both incur large performance degradation relative to baselines; neither transfers from single-image tasks.
  • SFT on existing multi-image data: yields only limited gains (detailed numbers in appendix).

Limitations

  • Human-expert baseline measured on only 52 problems, limiting statistical power for human–machine gap estimates.
  • Mathematics sub-domain contains only geometry-heavy topics (circle geometry, area/perimeter, transformations, tessellations, etc.); broader mathematical Olympiad coverage (number theory, combinatorics, algebra) is absent.
  • Chemistry image count is highest (avg. 4.00), but biology is similar to math (avg. 3.07); variance across disciplines may confound cross-discipline comparisons.
  • Rationale generation relied on Gemini-2.5-pro-thinking; any systematic errors in its solutions propagate despite human verification.
  • SFT experiments are limited to InternVL3.5-8B and Qwen3-VL-8B; generalization to larger or different architectures is untested.
  • The information-equivalent single-image control (image concatenation) does not perfectly isolate all confounders (e.g., spatial layout changes after concatenation).

Relevance to Vision-Language Models

OMIBench directly stress-tests the cross-modal and cross-image reasoning frontier of LVLMs, a capability dimension that single-image benchmarks like OlympiadBench, MathVista, and MathVerse cannot evaluate. The finding that even the strongest models plateau near 50% while human experts exceed 80%—and that parameter scaling, think-with-image, and standard CoT strategies all fail to close the gap—suggests that current LVLM architectures lack the mechanisms needed for coherent information flow across image sequences. For researchers tracking VLMs, the work identifies cross-image association (29% of errors) and multi-step logical integration (41%) as the two dominant failure modes, motivating architectural changes beyond prompt engineering or test-time compute. The diagnostic showing that activated (not total) parameter count predicts performance on MoE models is an actionable signal for model design targeting multi-image reasoning.