Skip to content

MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Provenance-aware multimodal browsing benchmark requiring genuine vision reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✩ Ask Claude

TL;DR

MMSearch-Plus is a 311-task benchmark for multimodal browsing agents that enforces genuine visual reasoning by requiring agents to extrapolate from fine-grained spatial and temporal cues in images to out-of-image facts via iterative image–text retrieval. The best evaluated system (o3 with Full Rollout + SoM) achieves only 36.0–37.6% accuracy, far below human performance on many categories, establishing a hard open challenge for agentic MLLMs.

Problem

Existing multimodal browsing benchmarks (e.g., MMSearch) can be largely solved by text-only heuristics: a strong image-search engine retrieves pages whose surrounding text directly answers the question, so fine-grained visual reasoning is bypassed. Meanwhile, text-only benchmarks like BrowseComp are far harder than their multimodal counterparts, revealing a difficulty gap. MMSearch-Plus closes this gap by requiring sustained, part-based visual reasoning under noisy retrieval that cannot be shortcut with a single image-search call.

Method

Dataset construction (Spatial–Temporal Extrapolation). Questions are seeded from real-world events (video clips, arXiv figures) containing subtle visual signals—micro-text, broadcast overlays, uniform variations—that jointly imply facts absent from the prompt. Spatial extrapolation asks about off-screen participants or occluded entities; temporal extrapolation asks about events immediately before or after the depicted moment. Adversarial filtering removes questions solvable by GPT-4o/GPT-5/Gemini-2.5-Pro without retrieval; images are blurred/masked to prevent shortcutting. 311 tasks across 8 primary and 43 secondary categories; 69.8% labeled Difficult.

Agent framework. A model-agnostic web agent built on SerpApi interleaves text search (up to 20 rounds, 3–5 queries per round), image search (top-10 results cached and summarized per image), and a Set-of-Mark (SoM) zoom-in module. SoM provides human-verified bounding boxes overlaid on images; the agent can crop sub-regions and launch targeted image searches from them. Retrieved pages are summarized by Gemini into task-conditioned semantic summaries and provenance-linking evidence, compressed into a threaded conversation state for long-horizon reasoning.

Evaluation. LLM-as-a-judge (GPT-4o) against a set of acceptable answer variants; manual inspection confirmed full agreement with human judgment.

Key Contributions

  • MMSearch-Plus benchmark: 311 tasks requiring fine-grained visual reasoning with spatial–temporal extrapolation, resistant to parametric shortcuts and direct image search.
  • Spatial–Temporal Extrapolation curation procedure: principled method to construct questions whose answers are contextually implied but physically absent from the image, forcing iterative hypothesis-test-refine behavior.
  • Model-agnostic SoM agent framework: integrates text search, image search, and region-level zoom-in/sub-image search with provenance tracking and caching.
  • MMSearchâș_lite: a 239-task subset verified to be unsolvable without retrieval by any tested model, enabling clean measurement of tool-use behavior decoupled from parametric memorization.
  • Behavioral analysis: Markov transition probabilities between tool types, stepwise tool-distribution profiles, and scene-text modality analysis across closed- and open-source MLLMs.

Results

  • Best end-to-end accuracy: o3 Full Rollout + SoM = 37.6% (vs. human with browser = 22.8% average, but humans outperform models in Sports/Vlog categories).
  • SoM gains over Full Rollout: o3 +1.6 pp, Gemini-2.5-Pro +3.9 pp, Qwen-2.5-VL-72B +1.0 pp.
  • Without search: Qwen 0.0%, GPT-5 10.3%, Gemini 10.6%, o3 15.1% — confirming benchmark resists parametric shortcuts.
  • Image Search alone: o3 19.3%, GPT-5 16.4%, Gemini 16.4%, Qwen 13.5%.
  • Full Rollout: o3 36.0% (+20.9 over no-search), Gemini 23.8% (+13.2), Qwen 6.1% (regresses -7.4 from image-search).
  • On MMSearchâș_lite (239 retrieval-required tasks), best is o3 Text Search at 31.4%; qualitative trends hold, ruling out memorization as the main driver.
  • Easy-split regression: o3 loses 7.4 pp from Image Search to Full Rollout; manual inspection of 10 sampled cases shows 9/10 failures are due to the model never invoking image search during full rollout (tool-selection miscalibration, not over-retrieval).
  • Qwen produces 421 invalid image-search tool calls across 45 tasks, inflating retries without quality gain.

Limitations

  • Ceiling is low but human baseline is also limited: annotators capped at 20 minutes per task with a standard browser; the human score (22.8%) may underestimate true human capability.
  • Temporal drift: newer closed-source models with more recent training data progressively make previously-hard questions solvable without retrieval; requires ongoing benchmark refreshes.
  • SoM bounding boxes are human-verified, not automatically generated, limiting scalability and real-world applicability of the zoom-in pathway.
  • Qwen tool-use instability: open-source models produce high rates of malformed tool calls, suggesting the benchmark disproportionately stresses instruction-following fidelity alongside visual reasoning.
  • Text search dominates: most models default to OCR + text search over image search for scene text, meaning the image-search pathway is underexplored even when available.
  • 311 tasks is relatively small; category-level subcounts can be as few as 31 items (Games), making fine-grained comparisons noisy.

Relevance to Agentic AI / LLM Agents

MMSearch-Plus directly advances evaluation methodology for multimodal web agents by exposing a concrete failure mode — tool-selection miscalibration — that existing benchmarks mask: agents achieve high accuracy via text heuristics rather than genuine visual grounding. The SoM zoom-and-retrieve framework demonstrates a concrete architectural primitive (region-seeded retrieval with provenance tracking) that improves multi-step agentic reasoning. The Markov tool-transition analysis offers a behavioral lens for diagnosing agent policy quality beyond aggregate accuracy, which is directly actionable for training and RLHF on agentic tool use. The benchmark complements text-only hard browsing tasks (BrowseComp) and establishes that multimodal agentic search requires qualitatively different capabilities — iterative hypothesis formation, fine-grained visual grounding, and cross-modal evidence integration — that current frontier models have not solved.