Skip to content

MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✩ Ask Claude

TL;DR

MMSearch-Plus is a 311-task multimodal browsing benchmark that enforces genuine visual reasoning by requiring agents to extract fine-grained spatial/temporal cues and propagate them through iterative image–text retrieval under noisy conditions. It pairs the benchmark with a model-agnostic agent framework featuring a Set-of-Mark (SoM) zoom-and-retrieve module. The best system (o3 + SoM) reaches only 36.0–37.6% accuracy, exposing large gaps versus humans and highlighting tool-use miscalibration as the primary failure mode.

Problem

Existing multimodal browsing benchmarks (e.g., MMSearch) are too easy: a strong image-search engine retrieves pages whose surrounding text directly contains the answer, letting unimodal LLMs bypass vision entirely. The benchmarks do not elicit noisy-retrieval cross-validation, exhaustive part-based visual reasoning, or long tool-augmented chains—capabilities required by real-world tasks. Meanwhile, text-only hard benchmarks (BrowseComp) score MLLMs below 2%, showing multimodal benchmarks lag far behind appropriate difficulty.

Method

Spatial–Temporal Extrapolation (STE): Questions are seeded from real-world events (video keyframes, arXiv figures) such that answers require inferring out-of-frame facts—off-screen participants, scores from adjacent rounds, dates of broadcasts—by combining subtle in-image cues (micro-text, broadcast overlays, layout signatures) with iterative retrieval. This naturally expands the intermediate search space and resists parametric shortcuts.

Data pipeline: Annotators select keyframes that (i) contain noisy or vague perceptual cues, (ii) depict unfamiliar entities, and (iii) cannot be solved by direct Google image search. Adversarial filtering removes questions answerable by GPT-4o/GPT-5/Gemini-2.5-Pro without retrieval via cross-validation and image perturbation (blurring).

Agent framework: A model-agnostic harness wraps any MLLM with: (a) SerpApi text search (top-5 pages, up to 20 rounds, 3–5 queries per round), (b) SerpApi image search (top-10 results, cached per benchmark image), (c) a Set-of-Mark (SoM) module providing human-verified bounding boxes that the agent can zoom into and use as sub-image retrieval queries. Webpage summaries are compressed by Gemini into web_info and related_info fields to fit limited context windows. A threaded conversation state tracks tool calls, cropped views, summaries, and hypotheses across turns.

Evaluation: LLM-as-a-judge (GPT-4o) compares predictions against acceptable-answer sets; manual inspection confirmed 100% agreement with human judgment.

Key Contributions

  • MMSearch-Plus benchmark: 311 tasks across 8 primary / 43 secondary categories, difficulty-split (30.2% easy / 69.8% hard), 441 unique images, answers averaging 3.7 words.
  • Spatial–Temporal Extrapolation: principled curation methodology that constructs questions whose answers are physically absent from the image, requiring agents to reason beyond the depicted frame and moment.
  • Model-agnostic agent harness: integrates text search, image search, and SoM zoom-in into a unified, cacheable, threaded multi-round framework compatible with any MLLM.
  • MMSearchâș_lite: a 239-task internal-knowledge-free subset for isolating genuine retrieval behavior from parametric memorization.
  • Behavioral analysis: Markov transition matrices over tool types, stepwise tool distributions, and OCR-vs-image-search pathway analysis reveal systematic tool-use patterns and failure modes across models.

Results

  • Best end-to-end accuracy: o3 Full Rollout + SoM = 37.6%; human annotators with browser = 22.8% (humans outperformed on most categories; o3 outperformed humans on Geography without search: 31.2% vs. 20.3%).
  • SoM gains over Full Rollout: +3.9 pp (Gemini-2.5-Pro), +1.6 pp (o3), +1.0 pp (Qwen-2.5-VL-72B).
  • Without-search baselines: o3 15.1%, Gemini 10.6%, GPT-5 10.3%, Qwen 0.0%—confirming resistance to parametric shortcuts.
  • Full Rollout vs. no-search gains: o3 +20.9 pp, Gemini +13.2 pp; Qwen regressed −7.4 pp from image-search to full rollout due to 421 invalid tool calls across 45 tasks.
  • On MMSearchâș_lite (retrieval-required only): best result is o3 text-only rollout at 31.4%, with most models near the diagonal of full vs. lite performance—confirming trends reflect tool use, not memorization.
  • Easy-split accuracy did not improve from Image Search to Full Rollout for o3; 9/10 sampled failure cases showed the model skipping image search entirely in the full-rollout setting.

Limitations

  • Small scale: 311 tasks is modest; category-level estimates carry high variance.
  • Temporal drift: as closed-source models absorb newer training data, previously retrieval-requiring questions become solvable from memory, necessitating ongoing benchmark refresh.
  • SoM requires human annotation: bounding boxes are manually verified per image, limiting scalability.
  • Caching introduces distribution shift: image-search results are cached per benchmark image rather than re-queried per agent run, which may not reflect live web dynamics.
  • Search capped at 20 rounds: truncates reasoning on hardest tasks; Qwen's cap set to 10 due to pilot results, making cross-model comparison asymmetric.
  • Error analysis is partial: failure categorization is described as ongoing (section ends mid-sentence in the provided text).
  • Single summarizer: Gemini is used as the fixed page-summarization model even when Gemini is also the agent under evaluation, introducing potential distribution mismatch.

Relevance to Harnesses / Meta-Harnesses

MMSearch-Plus is itself a reusable agent harness: the search-agent framework (tool dispatch, caching layer, SoM module, threaded state, LLM-as-judge evaluation) is model-agnostic and designed to wrap any MLLM, making it a direct example of a meta-harness that orchestrates heterogeneous tools (text search, image search, zoom-in, page summarization) around an arbitrary backbone. The caching and context-compression design (dual-field webpage summaries, per-image search-result caches) addresses the practical engineering problem of running long multi-step agent evaluations at scale without re-querying APIs per run. The behavioral instrumentation—Markov transition matrices over tool types, stepwise tool-use distributions, trajectory-level correctness analysis—constitutes a harness telemetry layer that mirrors the observability concerns central to meta-harness research. For researchers building or evaluating harnesses, this paper provides concrete design patterns for tool-dispatch, state management, and evaluation infrastructure in multimodal agentic settings.