Skip to content

UIS-Digger: Towards Comprehensive Research Agent Systems for Real-world Unindexed Information Seeking

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

UIS-Digger addresses the blind spot of Unindexed Information Seeking (UIS)—information hidden behind dynamic interactions, embedded files, or uncrawled pages—which is neglected by every major web-agent benchmark. The paper contributes both UIS-QA (110 expert-annotated QA pairs) and a multi-agent framework that achieves 27.27% on UIS-QA using a ~30B backbone, beating systems built on O3 and GPT-4.1.

Problem

Existing LLM-based research agents (and their benchmarks: GAIA, BrowseComp) exclusively evaluate Indexed Information Seeking (IIS)—information retrievable via standard search-engine crawls. A formal gap exists: many real-world answers require interacting with dynamic webpage widgets, downloading files, or traversing deeply nested pages that search engines never index. No prior benchmark captures this, and no agent framework is explicitly designed or trained for it.

Method

UIS-Digger is a four-agent hierarchical framework orchestrated by a Planner that decomposes queries and dispatches to three specialist agents:

  • Web Searcher: issues parallel search-engine queries and shallow crawls; delegates to Web Surfer or File Reader when indexed results are insufficient.
  • Web Surfer: operates a live browser (click, scroll, type, select, form-submit, download, screenshot) in dual-mode: textual by default, switching to visual (screenshot) only when needed. Both modes share a single browser state and memory, eliminating synchronization overhead.
  • File Reader: processes downloaded PDF/XLSX/DOCX files chunk-by-chunk when content exceeds the context window.

The inner LLM is fine-tuned in two stages. First, SFT cold start: a powerful teacher model \(X^*\) generates one trajectory per training question at temperature 0; a judge filters for correctness and non-triviality; passing trajectories train \(X^s\). Second, RFT bootstrapping: \(X^s\) generates four trajectories per question at temperature 0.4; difficulty-reweighted reject sampling (harder questions retain more trajectories) produces final model \(X^r\).

Training data is synthesized from (a) real-world website explorations across 100+ domains and (b) three virtual websites simulating flight booking and data dashboards to specifically target failure modes in interactive widget manipulation (date pickers, radio buttons, filter graphs).

Key Contributions

  • Formal definition of Unindexed Information (\(\text{UI} = \{x \mid x \in P \setminus \tilde{\text{II}}\}\)) distinguishing UIS from IIS, with approximation-aware formalism grounded in practical agent behavior.
  • UIS-QA: first benchmark dedicated to UIS; 110 QA pairs with objectivity, authoritativeness, static-nature, and verifiability constraints; cross-validated by three annotators + automated LLM filter + offline DeepSeek-R1 inner-knowledge filter.
  • UIS-Digger: multi-agent system with dual-mode memory-shared browser and file reader, achieving state-of-the-art 27.27% on UIS-QA.
  • Two-stage SFT+RFT training pipeline over synthesized real-world and simulated trajectories, with difficulty-reweighted reject sampling.
  • Detailed failure analysis identifying two root causes of poor UIS performance: insufficient action space and weak foundation models.

Results

  • UIS-Digger (PanGu-38B / Qwen3-32B): 27.27% on UIS-QA; 50.5% / 47.6% on GAIA; 32.5% / 32.5% on BrowseComp-zh.
  • Best competing baseline (Memento, O3+GPT-4.1): 25.45% on UIS-QA (53.9 pp drop from its 70.9% GAIA score).
  • Tongyi-DR: 23.6% on UIS-QA vs. 70.9% on GAIA—a 47.3 pp drop.
  • Direct API inference baselines (DeepSeek-V3.1, Claude-Sonnet-4, GPT-5): 1.8%, 2.7%, 0.9% on UIS-QA.
  • Commercial systems (GLM-4.5, Doubao, Gemini-2.5-pro): 11.8%, 11.8%, 4.5% on UIS-QA.
  • UIS-Digger is the only system to outperform O3-based multi-agent frameworks on UIS-QA while remaining competitive on GAIA/BrowseComp.

Limitations

  • UIS-QA contains only 110 samples, limiting statistical power; 84/110 are in Chinese, restricting generality.
  • Benchmark excludes login-gated or CAPTCHA-protected pages, missing a significant portion of real unindexed content.
  • Benchmark answers are required to be static, which excludes inherently dynamic UIS scenarios (live dashboards, real-time data).
  • The formal UIS definition uses an approximation (finite search queries, top-\(k\) results) that does not fully capture the true indexed/unindexed boundary.
  • Training data is synthesized and may not cover the full diversity of real-world UIS domains.
  • Performance remains low (27.27% ceiling) even for the best system, indicating the problem is far from solved.

Relevance to Harnesses / Meta-Harnesses

UIS-Digger is a textbook example of a research agent harness: a Planner meta-agent dynamically decomposes tasks and routes subtasks to three specialist sub-agents, each with its own tool suite, forming a hierarchical multi-agent pipeline analogous to orchestrator/worker meta-harness patterns. The two-stage SFT+RFT training pipeline is itself a meta-harness for agent self-improvement—the harness runs itself to generate trajectories, filters them, and trains the inner model iteratively. The dual-mode browser strategy (text-first, visual-on-demand with shared state) demonstrates a concrete design principle for harnesses that manage heterogeneous tool modalities without synchronization overhead. For researchers building meta-harnesses, UIS-Digger illustrates how difficulty-reweighted trajectory sampling and simulated-environment data augmentation can be integrated into the harness training loop to target identified capability gaps.