UNIBROWSE: A Data-to-Agent Framework for Multimodal BrowseComp

🕒 Published (v1): 2026-07-12 04:13 UTC · Source: Arxiv · link

Why this paper was selected

Data-to-agent framework for multimodal web browsing; compositional tool-use and perception integration

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

UNIBROWSE is a unified data-to-agent pipeline that generates multimodal web-browsing training data covering all three information-flow patterns (text-only, image-to-text, text-to-image) and trains a 35B agent via SFT + exploration-aware RL. It introduces an exploration degree metric to filter low-signal RL instances, yielding state-of-the-art performance on multimodal BrowseComp benchmarks (54.4 average accuracy across five benchmarks), surpassing GPT-5 (42.9) and Gemini-2.5 Pro (44.8) in agent-workflow settings.

Problem

Existing multimodal BrowseComp data pipelines cover only the image-to-text pattern (visual clue → textual answer), leaving the text-to-image pattern (textual reasoning → visual verification) unaddressed. Additionally, prior work uses either knowledge-graph-only construction (low web diversity) or random-walk construction (high noise), with no mechanism to filter RL instances by their learning-signal richness — resulting in wasted compute on low-contrast, uniformly-solved trajectories.

Method

The pipeline has three stages:

1. Textual KG with web expansion: From a knowledge graph \(G_{KG}\), extract a layered evidence subgraph \(G_T(s)\) around seed entity \(s\), where layer \(L_k\) depends on \(L_0, \ldots, L_{k-1}\) and the final layer \(L_L\) contains the target answer. Each entity node \(v\) is enriched with a verified attribute set \(\mathcal{A}(v)\) via live web retrieval + LLM extraction with cross-source corroboration, yielding \(C_T(s) = (G_T(s), \{\mathcal{A}(v)\}_{v \in G_T(s)})\).

2. Unified multimodal pattern expansion: The same subgraph is materialized into up to three patterns. Image-to-text: attach an image to \(L_0\) entities with retrievable visual representations (\(U^{i2t}(L_0)\)). Text-to-image: attach a visually distinctive image to the answer entity \(a \in L_L\), verified by a multimodal judge for entity identifiability, unique visual attribute absent from text, and image necessity (\(U^{t2i}(a)\)). Patterns are not mutually exclusive; a single question can carry both.

3. Exploration-degree filtering for RL: For question \(q\) with key evidence steps \(\pi\), sample \(B\) rollout trajectories and collect the successful subset \(Y^+(q)\). Define normalized exploration length \(\ell(\tau_b, \pi) = \frac{\text{\# browsing steps in } \tau_b}{|\pi|}\) and set $\(E(q) = \operatorname{Var}_{\tau_b \in Y^+(q)} \ell(\tau_b, \pi).\)$ High \(E(q)\) indicates that correct trajectories vary substantially in browsing effort (direct vs. reformulation-heavy paths), providing richer contrastive RL signal. Only the top-30% by \(E(q)\) are retained for RL (~10K QA pairs); \(E(q)\) guides selection only, not reward.

Agent training: cold-start SFT on 8K tool-use trajectories to establish action grammar, then GRPO-based RL on 10K exploration-rich QA pairs. The agent uses a SEARCH tool (text_only / image_to_text / text_to_image modes via SerpAPI) and LINKSUMMARY (auxiliary Qwen3-30B-A3B page reader) in a ReAct-style loop with typed serialization (<think>, <tool_call>, <tool_response>, <answer>).

Key Contributions

• First unified data pipeline generating all three multimodal information-flow patterns (text-only, image-to-text, text-to-image) from shared knowledge-graph subgraphs.
• Hybrid construction paradigm: KG-structured reasoning paths augmented with live web retrieval for real-world fidelity.
• Exploration degree metric \(E(q)\) — a variance-based signal over successful rollout lengths — enabling principled RL instance selection tailored to long-horizon BrowseComp tasks.
• 35B UNIBROWSE agent (Qwen3.5-35B-A3B backbone) achieving SOTA among open-source models on five multimodal browsing benchmarks.

Results

• Average (5 benchmarks): 54.4 (SFT+RL) vs. 43.9 (base Qwen3.5-35B-A3B agent workflow) — +10.5 points.
• Surpasses GPT-5 agent workflow (42.9), Gemini-2.5 Pro (44.8), Gemini-2.5 Flash (41.3), and Kimi K2.5 (50.8).
• MM-BrowseComp (only benchmark covering all three patterns): 28.2 (SFT+RL) vs. 17.1 (base) vs. 14.8 (Gemini-2.5 Pro agent workflow).
• BC-VL: 57.8 vs. 50.4 (base).
• FVQA: 76.7 vs. 71.0 (base), exceeding Doubao Seed 2.0 Pro (77.5) and Gemini-3.1 Pro (78.3) agent workflows.
• SFT alone (no RL): 54.4 average; adding RL pushes further on MM-BC (21.8 → 28.2) and BC-VL (54.4 → 57.8).
• Best prior open-source: Vision-DeepResearch-30B-A3B at 49.6 average; UNIBROWSE exceeds by 4.8 points.

Limitations

• Text-to-image filtering requires a multimodal judge plus image retrieval for every candidate answer entity — the filtering pipeline cost is non-trivial at scale.
• Exploration degree is computed by sampling \(B\) rollouts per question using a base agent, which is expensive; the exact \(B\) and the compute impact are not detailed in the main text.
• The MM-BrowseComp subset used excludes video reasoning and map-specific tool questions, limiting comparability to the full benchmark.
• The pipeline relies on SerpAPI and live web retrieval; reproducibility and temporal consistency of constructed data depend on external service availability.
• The 35B parameter scale limits deployment scenarios where inference cost is constrained; no smaller-model ablation is reported.

Relevance to Harnesses / Meta-Harnesses

UNIBROWSE is itself a data-construction harness — a multi-stage pipeline that orchestrates KG traversal, live web retrieval, VLM judging, quality filtering, and trajectory generation as a unified system feeding into downstream agent training. The exploration degree mechanism functions as a meta-level harness signal: it evaluates the output of a rollout harness (multiple agent trajectories per question) and uses that signal to govern which instances are admitted to the expensive RL stage, effectively making the data pipeline self-regulating. The typed serialization protocol and tool-environment abstraction (SEARCH / LINKSUMMARY) constitute a reusable agent harness that enforces action grammar independently of the task content. This work is directly relevant to lines of research on how to design harnesses that generate training data for other agents — a form of meta-harness where one orchestration system produces the curriculum for another.