Skip to content

SearchEyes: Towards Frontier Multimodal Deep Search Intelligence via Search World Simulation

🕒 Published (v1): 2026-07-07 07:43 UTC · Source: Arxiv · link

Why this paper was selected

Unified training-data, search-env, and reward for multimodal search agents via world simulation — end-to-end recipe

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SearchEyes trains multimodal search agents for multi-hop visual+textual reasoning by using a typed knowledge graph as a unified backbone for data synthesis, environment simulation, and RL reward signals simultaneously. It introduces Perception-Knowledge Chains (PKC) for constrained multi-hop question generation and Hop-Anchored Policy Optimization (HaPO) for step-level credit assignment without a separate process reward model. SearchEyes-27B outperforms the strongest open-source multimodal search baseline by 6.2 points on average across six benchmarks.

Problem

Existing multimodal search agent pipelines suffer from a three-way structural disconnect: (1) training data pipelines leverage knowledge graphs for question synthesis but discard intermediate entity metadata (path structure, entity IDs, relation types) after synthesis; (2) without shared structural information, training environments depend on external search engines, introducing irreproducibility and API cost; (3) RL reward signals are computed only at the trajectory level from final answers, making credit assignment prohibitively sparse for long-horizon multi-hop chains where completion probability decays exponentially with hop count.

Method

Unified Knowledge Graph Backbone. A typed KG is built from Wikidata5M (relation triples), Wiki6M (entity-level text), and Wikipedia images. Entities are retained only if present in all three sources; the graph is typed over four semantic domains \(\mathcal{T} = \{\text{Person, Work, Org, Geo}\}\) via predicate-to-domain mapping \(\phi: P \to \mathcal{T}\).

Perception-Knowledge Chains (PKC). Multi-hop paths are sampled via constrained random walks enforcing: (C1) strict P-K alternation—perception hops identify a visual entity from an image; knowledge hops search text—with \(\geq 2\) P-hops per path; (C2) a disambiguating constraint edge raising reasoning-graph treewidth to 2; (C3) semantic domain diversity requiring \(\phi(r_k) \neq \phi(r_{k+1})\) and \(\geq 3\) distinct domains; (C4) anti-shortcut filtering removing hub nodes, blacklisted predicates, and duplicate anchor-answer pairs. Questions are generated with three concealment constraints: visual anchoring (anchor entity name withheld), relation obfuscation (only a domain-level hint \(\tilde{d} = \phi(r_j)\) disclosed), and constraint obfuscation (disambiguating entity described relationally, not by name). The gold entity sequence \(e^* = (v_1, \ldots, v_{K+1})\) is retained as metadata throughout.

Self-Contained Search World. A deterministic retrieval environment over \(\mathcal{D} = \{\text{doc}(v) \mid v \in V\}\) uses hybrid BM25 + dense retrieval fused via RRF: $\(\text{score}_{\text{RRF}}(q, v) = \sum_{r \in \{\text{dense, BM25}\}} \frac{1}{\kappa + \text{rank}_r(q, v)}\)$ Agent tools: text_search, lookup, visual_search (image embedding cosine search), summarize, python_interpreter.

SFT with Privileged Conditions. An expert policy generates trajectories in a modified environment \(\tilde{E}\) where gold-entity retrieval scores are boosted by factor \(\beta > 1\), observations are denoised by an LLM summarizer, and trajectories are accepted via rejection sampling. Exported SFT data replaces denoised observations with raw observations; training masks observation tokens from the loss: $\(\mathcal{L}_{\text{SFT}} = -\sum_{\ell=1}^{L} M_{\text{gen}}(\ell) \cdot \log \pi_\theta(x_\ell \mid x_{<\ell})\)$

Hop-Anchored Policy Optimization (HaPO). After SFT, RL fine-tuning reuses the gold entity sequence \(e^*\) from PKC as step-level anchors. Rather than assigning a uniform trajectory-level advantage to all tokens, HaPO groups tokens by hop \(k\) (tokens in the same rollout at hop \(k\) form a group), computes step-level advantages via within-hop variance, applies fatal-aware masking to suppress clearly incorrect intermediate states, and blends hop-level and step-level signals for the policy update—without training a separate process reward model.

Key Contributions

  • Unified search world simulation: typed KG (Wikidata5M + Wiki6M + Wikipedia images) simultaneously defines training data, retrieval environment, and step-level reward anchors—eliminating the data–environment–reward disconnect
  • PKC synthesis: constrained multi-hop path sampling with P-K alternation, disambiguation constraints (treewidth \(\leq 2\)), domain diversity, and multi-level anti-shortcut filtering; all entity metadata retained for downstream training
  • HaPO: step-level credit assignment by reusing KG-derived hop anchors; no separately trained process reward model required
  • Agentic post-training pipeline: SFT with retrieval-boosted privileged environment + observation denoising, followed by HaPO
  • VisSearch Bench: new benchmark for multi-hop visual search evaluation (Appendix A)

Results

  • SearchEyes-27B achieves +6.2 points average over the strongest open-source multimodal search baseline across six multimodal knowledge-intensive benchmarks
  • SearchEyes-27B: 82.4 on MMSearch, 79.1 on FVQA—competitive with Gemini-3.1-Pro (proprietary)
  • SearchEyes-9B matches 30B-scale baselines (e.g., OpenSearch-VL-30B at ~59.8%) with 3.3Ă— fewer parameters
  • Baselines compared: OpenSearch-VL-8B/32B, ODE-8B/30B, SenseSearch-8B, DeepEyes-V2-7B, WebWatcher-7B/32B, MMSearch-R1-7B, Visual-ARFT-7B, Qwen3.5-9B/27B (Agentic)

Limitations

  • The search world is closed over Wikidata5M entities; questions and evaluation are Wikipedia-derived, limiting generalization to open-web or domain-specific retrieval scenarios
  • Retrieval boost during SFT trajectory synthesis may create a distributional gap if gold-entity documents are absent in the un-boosted environment at test time
  • PKC synthesis constraints (alternation, treewidth \(\leq 2\), domain diversity) define a specific question distribution that may not cover all real-world multi-hop question types
  • Paper text is truncated; the full limitations and ablation sections are not available for review

Relevance to Agentic AI / LLM Agents

SearchEyes directly addresses the core challenge of long-horizon agentic RL: sparse reward and credit assignment degradation in multi-turn tool-use trajectories, a bottleneck shared across all RL-trained agent systems (web agents, computer-use, code agents). The co-design of data synthesis, environment simulation, and reward signal from a single structured knowledge source is a general architectural principle applicable beyond multimodal search. HaPO's approach—deriving step-level reward anchors from the same structure used for question generation, without a separate PRM—offers a scalable alternative to process reward models for agentic RL. The self-contained, deterministic search world addresses the reproducibility problem that plagues agentic training pipelines relying on live external APIs.