Skip to content

MMSearch-Plus: Benchmarking Provenance-Aware Search for Multimodal Browsing Agents

🕒 Published (v1): 2025-08-29 09:58 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MMSearch-Plus is a 311-task benchmark requiring multimodal browsing agents to perform genuine visual reasoning via iterative image–text retrieval, resisting text-only shortcuts through Spatial–Temporal Extrapolation. It pairs the benchmark with a model-agnostic agentic framework augmented by a Set-of-Mark (SoM) module for provenance-aware zoom-and-retrieve. The best system (o3, Full Rollout + SoM) achieves 36.0–37.6% accuracy, well below the human baseline of 22.8% (browser-assisted), indicating substantial headroom.

Problem

Existing multimodal browsing benchmarks (e.g., MMSearch) allow strong image search engines to retrieve answer-bearing text, letting unimodal LLMs bypass visual reasoning entirely. Tasks degenerate into image-to-source cross-validation, with fine-grained visual reasoning playing a marginal role. Text-only browsing benchmarks (e.g., BrowseComp) are substantially harder, revealing an inverse difficulty gap: multimodal benchmarks are far easier than their text-only counterparts despite real-world multimodal tasks demanding deeper reasoning.

Method

Spatial–Temporal Extrapolation: Questions are constructed so answers require reasoning beyond the depicted frame. Spatial extrapolation asks about off-screen or occluded entities; temporal extrapolation probes events immediately before/after the depicted moment (e.g., the next goal, subsequent episode segment). Visual cues are deliberately subtle: micro-text, broadcast overlays, uniform variations, layout signatures.

Data collection: Keyframes from video platforms and arXiv figures are screened so they (i) contain perceptually challenging information, (ii) feature unfamiliar entities requiring retrieval, and (iii) cannot be resolved by direct image search. Adversarial filtering applies cross-model validation, image perturbation (blurring/masking), and iterative refinement to eliminate questions solvable from parametric memory.

Agent framework: A model-agnostic web agent with SerpApi-backed text and image search. Webpages are summarized by a secondary MLLM (Gemini) into task-conditioned web_info and related_info fields to compress context. Image-search results are cached per benchmark image. The Set-of-Mark (SoM) module supplies human-verified bounding boxes overlaid on input images, enabling (a) zoom-in on marked subregions and (b) sub-image search to seed fine-grained retrieval queries. Search is capped at 20 rounds; conversation state threads tool calls, crops, summaries, and hypotheses across turns.

Key Contributions

  • A 311-task multimodal browsing benchmark (8 primary, 43 secondary categories; 69.8% "difficult") requiring evidence propagation through noisy iterative retrieval
  • Spatial–Temporal Extrapolation curation methodology that enforces reasoning beyond the depicted frame
  • A model-agnostic agentic framework with SoM zoom-and-retrieve, enabling ablation of cropping and sub-image search strategies
  • Comparative analysis of tool-use behavior across closed-source (o3, GPT-5, Gemini-2.5-Pro) and open-source (Qwen-2.5-VL-72B) MLLMs
  • MMSearch\(^+_\text{lite}\): a 239-task subset confirmed unsolvable without search, used to decouple internal knowledge from retrieval behavior

Results

  • Best overall: o3 Full Rollout + SoM: 37.6% accuracy; o3 Full Rollout alone: 36.0% (+20.9 pp over no-search)
  • Human (browser, 10–20 min): 22.8%
  • SoM gains over Full Rollout: Gemini +3.9 pp (23.8→27.7), o3 +1.6 pp, Qwen +1.0 pp
  • No-search baseline: Qwen 0.0%, GPT-5 10.3%, Gemini 10.6%, o3 15.1% — confirms parametric shortcuts are suppressed
  • Image Search: moderate gains (+4.2–13.5 pp), but insufficient without multi-hop text retrieval
  • MMSearch\(^+_\text{lite}\): o3 Text Search achieves 31.4%; full rollout + SoM does not consistently outperform text-only on this retrieval-hard subset
  • Easy split anomaly: o3 loses 7.4 pp absolute on easy tasks when switching from image-search to full rollout, driven primarily by underuse of image search (9/10 sampled errors), not over-retrieval

Limitations

  • Human-verified SoM bounding boxes are manually provided, limiting scalability to automatic bounding-box generation for deployment
  • Benchmark covers only 311 tasks across 8 categories; imbalanced category coverage (Games category notably low)
  • Temporal drift: as frontier models incorporate newer data, tasks that required retrieval can become solvable from memory; benchmark requires ongoing refresh
  • Search capped at 20 rounds; Qwen performance plateaued at 10 rounds, suggesting open-source models may be under-evaluated under higher budgets
  • Error taxonomy (Section 5.2) is manually curated and single-label per case, limiting granularity

Relevance to Harnesses / Meta-Harnesses

MMSearch-Plus contributes a model-agnostic agent harness with explicit orchestration logic: caching, context threading, multi-round tool dispatching (text search, image search, zoom-in), and secondary MLLM summarization as a middleware layer — all architectural patterns central to meta-harness design. The SoM module demonstrates how a harness can inject structured, human-verified retrieval anchors (bounding boxes) to guide sub-agent tool calls, an example of harness-controlled attention steering. The benchmark's evaluation pipeline — LLM-as-judge, caching of MLLM summaries per image, per-round loop orchestration — is itself a harness that others building evaluation meta-harnesses for multimodal agents can adapt. The failure analysis revealing tool-use policy errors (e.g., skipping image search in full rollout) points to a key open problem for meta-harnesses: learned or rule-based tool-selection policies that override model-level miscalibration.