Skip to content

UIS-Digger: Towards Comprehensive Research Agent Systems for Real-world Unindexed Information Seeking

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Agents for unindexed non-web-indexed real-world information seeking

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Current LLM-based information-seeking agents rely exclusively on search-engine-indexed content, failing on a large class of real-world queries where the answer resides in dynamically rendered pages, deeply nested sites, or downloadable files. This paper defines this gap formally as Unindexed Information Seeking (UIS), introduces UIS-QA (110 expert-annotated QA pairs) as the first dedicated benchmark, and proposes UIS-Digger, a multi-agent framework with dual-mode browsing and a two-stage fine-tuned ~30B LLM backbone, achieving state-of-the-art 27.27% on UIS-QA.

Problem

Existing agent benchmarks (GAIA, BrowseComp) measure performance on questions answerable via standard search-engine retrieval. They do not evaluate whether agents can discover information that is not indexed—content behind interactive UI widgets (date pickers, filters, option selectors), embedded in downloadable files (PDF, XLSX, DOCX), or on pages excluded from crawlers. Even the best existing agents drop from >70% on GAIA to ~25% on UIS-QA, revealing a blind spot missed by the entire evaluation paradigm.

Method

Formal definition. Let \(P^{(E)}\) be the set of search-engine-indexed pages and \(\widetilde{II}\) the practically accessible indexed information (snippets + one-hop crawl from query results). UIS-QA tasks are defined as problems \((Q, C)\) requiring context \(C^{(U)}\) with \(|C^{(U)}| > 0\) such that \((Q, C^{(I)})\) alone cannot yield the correct answer \(z\).

UIS-Digger architecture. A four-agent system: - Planner decomposes the user query into subtasks and orchestrates the other agents. - Web Searcher issues concurrent search-engine queries and shallow crawls, delegating deeper navigation to the Web Surfer or File Reader. - Web Surfer operates a real browser (click, scroll, type, select, download, screenshot) using a dual-mode strategy: textual mode by default for efficiency, visual (screenshot) mode when textual parsing is insufficient. Both modes share a single memory and browser state. - File Reader processes downloaded PDFs, XLSXs, and DOCXs incrementally when content exceeds the context window.

Two-stage LLM training. 1. SFT cold start: A teacher model \(X^*\) generates one trajectory per training question at temperature 0; correct, non-trivial trajectories are retained via reject sampling to produce \(X^s\). 2. RFT bootstrapping: \(X^s\) generates 4 trajectories per question at temperature 0.4; difficult questions (fewer correct attempts) are upweighted. The resulting model \(X^r\) is the default backbone.

Training data comes from both real-world website exploration (100+ base homepages across government, product catalogs, code repos) and simulated interactive websites (flight booking, statistical dashboards) that expose widget-manipulation edge cases.

Key Contributions

  • Formal definition and problem taxonomy of Unindexed Information Seeking (UIS), distinguishing it from Indexed Information Seeking (IIS).
  • UIS-QA: 110 expert-annotated, cross-validated, statically stable QA pairs requiring unindexed sources; filtered against Google Search, an LLM oracle (z.ai), and offline LLM inner-knowledge checks.
  • UIS-Digger: a multi-agent framework with dual-mode (textual+visual) shared-memory browser, file reading, and parallel tool execution.
  • Two-stage SFT→RFT training pipeline with real + simulated trajectory data, yielding a ~30B model that outperforms O3- and GPT-4.1-backed systems on UIS-QA.
  • Failure-case analysis and behavior evolution study across training stages.

Results

  • UIS-Digger (Pangu-38B / Qwen3-32B): 27.27% on UIS-QA; 50.5% / 47.6% on GAIA; 32.5% / 32.5% on BrowseComp-zh — best on UIS-QA across all evaluated systems.
  • Best prior multi-agent baseline (Memento, O3 + GPT-4.1): 25.45% on UIS-QA vs. 70.9% on GAIA — a 53.9 pp drop.
  • Tongyi-DR (~70% GAIA): drops to 23.6% on UIS-QA (-47.3 pp).
  • Direct API inference (GPT-5, Claude Sonnet 4, DeepSeek-V3.1): all score ≤2.7% on UIS-QA.
  • Commercial systems (GLM-4.5, Doubao, Gemini-2.5-pro): 4.5–11.8% on UIS-QA.
  • WebSailor (ReAct, strong on IIS): 7.3% on UIS-QA.
  • UIS-Digger's ~30B backbone outperforms all systems including those with O3, demonstrating that action-space breadth and UIS-specific training matter more than raw model size.

Limitations

  • UIS-QA contains only 110 samples, limiting statistical power; 84 of 110 are Chinese-language questions, raising questions about cross-lingual generalizability.
  • The benchmark excludes login-gated content and CAPTCHA-protected pages by design, leaving a significant class of real-world UIS problems unaddressed.
  • UIS-QA covers static answers only; dynamically changing content (e.g., live statistics) is explicitly excluded.
  • Best accuracy is still only 27.27%, leaving substantial room for improvement; UIS remains largely unsolved.
  • Training data construction depends on UIS-Digger itself for trajectory generation, creating potential distribution bias.
  • The paper does not report latency or cost metrics, which are critical for practical deployment of deep browser-interaction agents.

Relevance to Agentic AI / LLM Agents

UIS-Digger directly advances the agentic information-seeking frontier by exposing a systematic blind spot in how agents are evaluated and trained: standard ReAct-over-search-engine pipelines structurally cannot solve a large category of real queries. The dual-mode browsing architecture (textual default, visual fallback, shared state) is a concrete design pattern applicable to any web-agent system aiming for comprehensive coverage. The two-stage SFT→RFT training recipe with difficulty-reweighted reject sampling provides a replicable methodology for bootstrapping specialized tool-use capabilities from a mid-sized model. For researchers tracking LLM agents, this work motivates expanding action spaces beyond search-and-crawl and rethinking benchmarks that inadvertently reward indexed-only retrieval.