Skip to content

MC-Search: Evaluating and Enhancing Multimodal Agentic Search with Structured Long Reasoning Chains

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Multimodal agentic search with structured long reasoning chains

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MC-Search is a benchmark for multimodal agentic RAG (MM-RAG) featuring 3,333 examples with long, step-wise annotated reasoning chains (avg. 3.7 hops) across five distinct reasoning topologies. It introduces process-level metrics beyond answer accuracy and the SEARCH-ALIGN fine-tuning framework. Benchmarking six leading MLLMs reveals systematic failures including over-/under-retrieval and modality-misaligned planning.

Problem

Existing MM-RAG benchmarks are limited to 1–2 hop retrievals with fixed retrieve-then-generate pipelines, lack step-wise trajectory annotations, and collapse multimodal evidence into a single textual channel. They cannot evaluate whether MLLMs can perform adaptive, long-horizon, cross-modal agentic reasoning in practice.

Method

Benchmark construction: Wikipedia-grounded multimodal knowledge clusters are used to generate ~21k candidate examples via Gemini-2.5-Flash, each specifying a reasoning graph \(G(Q,A) = \{(q_t, m_t, r_t, a_t)\}_{t=1}^T\) over five topologies: Image-Initiated Chain, Text-Initiated Chain, Text-Only Chain, Parallel Image-Text Fork, and Multi-Images Fork.

HAVE filtering: Each hop is evaluated for context utility \(\text{Util}(t) = \text{F1}(C) - \text{F1}(C \setminus r_t)\) and navigational necessity \(\text{Nav}(t) = \mathbf{1}[\text{Ent}(a_t) \cap \text{Ent}(q_{t+1:T}) \neq \emptyset]\). Hops with low utility and zero navigational role are pruned as redundant, reducing ~21k to 3,333 high-quality chains.

Agentic MM-RAG pipeline: At each step the MLLM generates a sub-query and selects one of three retrieval actions (text search, image search with text query, image search with input image), retrieves top-1 evidence via dense retrieval, generates a sub-answer, and iterates until termination.

SEARCH-ALIGN: HAVE-verified trajectories are augmented with Gemini-2.5-Flash-generated reasoning thoughts and converted to conversation-style dialogue traces for supervised fine-tuning, providing process-level (step-wise) rather than answer-only supervision.

Process-level metrics: (i) LLM-as-a-Judge (LJ) on four dimensions; (ii) Hit per Step (HPS) via maximum-weight bipartite matching between predicted and golden evidence steps: \(\text{HPS}(\hat{G}, G) = \frac{1}{|G|}|\{(t,t') \mid \hat{r}_{t'} = r_t\}|\); (iii) Rollout Deviation \(\text{RD}(\hat{G}, G) = |\hat{G}| - |G|\) measuring over/under-retrieval.

Key Contributions

  • First agentic MM-RAG benchmark with long (\(\geq\)4 hop), step-wise annotated reasoning chains across five reasoning topologies, verified for necessity and non-redundancy via HAVE
  • Three process-level metrics (LJ, HPS, RD) enabling fine-grained evaluation of retrieval and planning fidelity beyond final-answer accuracy
  • Unified agentic MM-RAG pipeline and comprehensive benchmarking of six MLLMs (proprietary + open-source), cataloguing eight characteristic error types
  • SEARCH-ALIGN: conversation-level, process-supervised SFT framework improving planning and retrieval fidelity in open-source MLLMs

Results

  • Dataset quality: average Gemini-2.5-Pro score of 4.87/5.0 on factual correctness, step necessity, clarity, and multimodal alignment
  • Benchmarking reveals all six tested MLLMs exhibit systematic over-retrieval (positive RD) and under-retrieval (negative RD), as well as modality-misaligned planning (choosing wrong retrieval modality)
  • SEARCH-ALIGN improves open-source MLLM performance on both answer accuracy and process-level metrics over base SFT (specific delta numbers were not included in the provided text excerpt)
  • Knowledge base scale: 389,750 images and 784,473 documents

Limitations

  • Dataset sourced entirely from Wikipedia, which may not capture the full diversity of real-world agentic search scenarios
  • HAVE filtering relies on Gemini-2.5-Flash/Pro as evaluators, inheriting potential model-specific biases even with the mitigation step using Qwen2.5-VL-7B
  • Top-1 retrieval per sub-query in the pipeline is a simplification; real agentic systems often use top-k or re-ranking
  • Specific quantitative SEARCH-ALIGN improvement numbers are not reported in the available text
  • Reasoning topology coverage is fixed to five predefined structures, which may not exhaust real-world graph diversity

Relevance to Agentic AI / LLM Agents

MC-Search directly addresses the agentic loop core to LLM agent research: iterative sub-goal decomposition, adaptive tool/modality selection, and multi-step evidence integration—extending the RAG paradigm from single-shot retrieval to a full agent trajectory. The HAVE filtering methodology and process-level metrics (HPS, RD) are broadly applicable beyond multimodal settings as principled tools for evaluating any agent that must follow a structured reasoning trajectory. The revealed failure modes (over-/under-retrieval, modality misalignment) are fundamental agent planning pathologies, and SEARCH-ALIGN's process-supervised SFT approach is a concrete recipe for improving agent faithfulness to intended execution plans. This work bridges the gap between agentic benchmark design and agent training, making it directly relevant to anyone building or evaluating LLM-based agents that interleave reasoning with tool use.