Skip to content

Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agents

🕒 Published (v1): 2026-05-11 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces a visual-native agent harness and On-policy Data Evolution (ODE), a closed-loop data construction framework for multimodal deep search agents. The harness addresses the gap that tool-returned images are typically treated as transient outputs rather than reusable state; ODE addresses the gap that training data synthesis is usually fixed rather than adaptive to the current policy's capability. Together, they lift Qwen3-VL-8B from 24.9% to 39.0% average accuracy across 8 benchmarks, surpassing Gemini-2.5 Pro's 37.9%.

Problem

Two orthogonal bottlenecks limit multimodal deep search: (1) existing agent harnesses discard intermediate visual evidence—images returned by search, browsing, or image manipulation tools cannot be passed as inputs to later tool calls, breaking cross-step visual evidence chains; (2) training data is constructed with a fixed recipe chosen before training begins, so the data cannot track the evolving capability of the target policy and cannot steer synthesis toward the policy's current learning frontier.

Method

Visual-Native Agent Harness. Nine tools (web search, image search, scholar search, visit/browse, visual search, zoom-in, rotation, flip, Python execution) operate over a shared workspace governed by an image bank reference protocol. Every image—initial task image and every tool-returned image—is registered under an addressable <image:N> handle. Any subsequent tool call can consume these handles, converting single-image interaction into a chained workflow with accumulated visual state.

On-policy Data Evolution (ODE). Data construction is parameterized by a fixed System Config and an editable Evolvable Config \(C_t\). Each round \(t\) runs two pipelines:

  • Forward curation: (i) Seed Proposal samples entity-image pairs across 11 domains, 4 capability profiles, and 4 difficulty levels, retaining only seeds with verifiable multi-source support; (ii) Web Exploration uses the nine harness tools to build evidence nodes; (iii) Graph Organization connects nodes into a multimodal evidence graph \(G\), enriching it with reasoning nodes (from Python/visit) and perception nodes (from zoom-in, rotation, flip, visual search); (iv) Task Curation traces a reasoning path through \(G\), synthesizes a candidate \((q, I_0, a)\), rewrites the question to deepen required reasoning, and stores tasks that pass answer unambiguity and no-shortcut checks into \(D_t^{\text{cand}}\).

  • Backward optimization: The rollout model \(m_t\) executes each candidate in the harness, producing trace \(\tau_i\). A Trace Analysis module scores \(\tau_i\) on shared rubric dimensions (Information Complexity, Visual Dependency, Shortcut Leakage, Verifiability) plus mode-specific dimensions—SFT adds Step Appropriateness, Tool Usage Quality, Tool Pattern Diversity; RL adds Capability Requirement, Difficulty Match, Learning Utility. Per-trace diagnoses \(\delta_i\) point failures to the forward stage to revise. Aggregated signal \(\Delta_t\) edits \(C_t \to C_{t+1}\), adjusting seed selection, exploration breadth, organization guidance, and difficulty weights. For SFT, \(m_t\) is a teacher model; for RL, \(m_t\) is the current policy so rollout difficulty directly measures proximity to the learning frontier.

Key Contributions

  • Visual-native agent harness with image bank reference protocol, making all tool-produced images persistently reusable as named handles across the trajectory.
  • On-policy Data Evolution (ODE): a closed-loop data generator that runs task synthesis → policy rollout → rubric-based trace analysis → config update across rounds, jointly supporting SFT teacher-trace curation and RL policy-frontier task curation.
  • Unified framework spanning the full training lifecycle (SFT then RL), with mode-specific rubric dimensions and update rules for each phase.
  • Empirical validation across 8 multimodal deep search benchmarks at 8B and 30B scale.

Results

  • Qwen3-VL-8B: 24.9% → 39.0% average (+14.1 pp) over 8 benchmarks after ODE-SFT+ODE-RL.
  • Qwen3-VL-30B: 30.6% → 41.5% average (+10.9 pp).
  • ODE-trained 8B agent (39.0%) surpasses Gemini-2.5 Pro in agent-workflow setting (37.9%) and WebWatcher-32B (24.4%).
  • Harness ablation (image-bank reuse removed): MMBC −4.9 pp, HLE-VL −2.9 pp, MMSearch+ −3.2 pp on ODE-8B-RL; benchmarks with higher secondary image-reuse rates show larger drops, confirming causality.
  • Difficulty shift in ODE-curated RL data: Hard+Expert share rises from 29.1% (SFT set) → 61.9% (ODE-8B) → 93.7% (ODE-30B-A3B), tracking policy capability.
  • Static synthesis baseline vs. ODE (matched 2K SFT / 4K RL budgets): ODE yields higher downstream accuracy and more tool-grounded SFT traces (more 4+ tool-image traces, more 2+ tool calls).

Limitations

  • Evaluated on a single model family (Qwen3-VL at 8B and 30B); generalizability to other VLM backbones is unverified.
  • ODE requires multiple rounds of full policy rollout for both forward and backward passes, imposing significant compute overhead relative to static synthesis.
  • The rubric dimensions and optimization strategy are hand-designed; their completeness and transferability to domains beyond the 11 covered topics is not analyzed.
  • Task curation and trace analysis rely on an LLM judge for answer verification and diagnosis, introducing potential noise in the feedback signal.
  • No analysis of how many ODE rounds are needed for convergence, or of diminishing returns.

Relevance to Harnesses / Meta-Harnesses

The paper is directly relevant as a concrete instantiation of the harness concept: the Visual-Native Agent Harness formalizes a shared execution environment with a typed state protocol (image bank), demonstrating how harness design choices propagate into measurable capability gains. More significantly, ODE is a meta-harness: it treats the data-generation pipeline itself as a configurable, evolvable system driven by rollout feedback—analogous to a loss function operating over the data generator rather than the model. This closed-loop architecture, where the harness both executes policy rollouts and surfaces diagnostic signals that mutate the generation configuration, is a template for self-improving training pipelines and extends the meta-harness concept from orchestrating agents to orchestrating the agent's own curriculum.