Skip to content

UIS-Digger: Towards Comprehensive Research Agent Systems for Real-world Unindexed Information Seeking

🕒 Published (v1): 2026-03-09 08:58 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

UIS-Digger addresses the blind spot of Unindexed Information Seeking (UIS)—information unreachable via standard search-engine crawls—by introducing both a dedicated benchmark (UIS-QA, 110 expert-annotated QA pairs) and a multi-agent framework that combines dual-mode browsing, file parsing, and a two-stage fine-tuned ~30B LLM. The system achieves 27.27% on UIS-QA, surpassing all baselines including those backed by O3 and GPT-4.1.

Problem

Existing LLM-based information-seeking agents are evaluated on benchmarks (GAIA, BrowseComp) that assume search-engine-indexed knowledge is sufficient. A large class of real-world queries requires accessing unindexed content: deeply nested webpages, dynamically rendered pages, embedded files (PDF/XLSX/DOCX), and pages requiring interactive widget manipulation. No benchmark or framework targeted this gap; even the best agents drop from >70% on GAIA to ~25% on UIS tasks.

Method

UIS-Digger is a four-agent system—Planner, Web Searcher, Web Surfer, and File Reader—orchestrated via a request-response message bus under the ReAct paradigm.

  • Dual-mode browsing: the Web Surfer maintains a single shared browser state and memory but switches dynamically between textual and visual (screenshot) modes, defaulting to text for efficiency and invoking vision only when needed.
  • File Reader: handles PDF, XLSX, DOCX; chunked reading when content exceeds context window.
  • Parallel tool execution: Searcher and Surfer can operate concurrently.

Training proceeds in two stages over synthesized QA pairs derived from real websites (100+ homepages) and virtual interactive sites (flight booking, data dashboards with synthetic JSON databases): 1. SFT cold-start: teacher model \(X^*\) generates one trajectory per question at temperature 0; only correct, non-trivial trajectories are retained for fine-tuning, producing \(X^s\). 2. RFT bootstrapping: \(X^s\) samples 4 trajectories per question at temperature 0.4; reject-sampling retains correct ones, re-weighted by difficulty (harder questions up-weighted); yields final model \(X^r\).

Formally, a question \(Q\) is a UIS problem if its required context satisfies \(|C^{(U)}| > 0\) and \((Q, C^{(I)}) \not\Rightarrow z\), where \(C^{(I)}\) is all practically accessible indexed information and \(C^{(U)}\) is the unindexed remainder.

Key Contributions

  • Formal definition of the Unindexed Information Seeking (UIS) problem, distinct from Indexed Information Seeking (IIS).
  • UIS-QA: first benchmark for UIS, with 110 rigorously filtered QA pairs spanning government sites, code repos, company reports, and product catalogs; filtering pipeline uses human annotators, z.ai auto-verification, and an offline LLM (DeepSeek-R1) to remove IIS-solvable and knowledge-answerable items.
  • UIS-Digger: multi-agent framework with dual-mode shared-memory browsing, parallel tool execution, and a two-stage (SFT→RFT) training pipeline; achieves 27.27% on UIS-QA.
  • Detailed failure-mode analysis and public release of the dataset.

Results

  • UIS-Digger (Pangu-38B / Qwen3-32B): 27.27% on UIS-QA; 50.5% / 47.6% on GAIA; 32.5% / 32.5% on BrowseComp-zh.
  • Strongest prior baseline (Memento, O3+GPT-4.1): 25.45% on UIS-QA, 32.2% on GAIA (without case bank).
  • Tongyi-DR (70.9% on GAIA): drops to 23.6% on UIS-QA—a 47.3 pp decline.
  • Memento (best GAIA agent, >70%): drops to 25.5% on UIS-QA—a 53.9 pp decline.
  • Direct-inference LLMs (GPT-5, Claude Sonnet 4, DeepSeek-V3.1): 0.9–2.7% on UIS-QA.
  • Commercial systems (GLM-4.5, Doubao, Gemini-2.5-pro): 4.5–11.8% on UIS-QA.
  • UIS-Digger's ~30B backbone outperforms all systems regardless of backbone size.

Limitations

  • UIS-QA is small (110 samples), with 84 questions in Chinese and only 26 in English, limiting multilingual generalizability.
  • Best system accuracy is only 27.27%, leaving substantial headroom; the benchmark may be too hard for meaningful differentiation among weaker systems.
  • Training data is synthesized and biased toward domains used during construction (public companies, government sites, etc.).
  • The formal UIS definition uses practical approximations that depend on the choice of search engine (Google Serper) and number of queries \(m\), making UIS/IIS classification agent-relative.
  • Interactive elements requiring CAPCHAs and login-gated content are explicitly excluded, missing a significant real-world UIS category.

Relevance to Harnesses / Meta-Harnesses

UIS-Digger is a concrete instantiation of a research-agent meta-harness: a multi-agent orchestration layer that routes subtasks (search, deep browse, file parse) to specialized sub-agents, collects and synthesizes evidence, and trains the underlying LLMs through a self-improving pipeline (generate trajectories → filter → SFT → RFT). The two-stage SFT→RFT training loop mirrors the self-bootstrapping pattern common in meta-harness designs, where the system iteratively improves its own trajectory quality. For researchers building harnesses for information retrieval or knowledge synthesis, the dual-mode browser architecture and the difficulty-reweighted RFT stage are directly transferable design patterns. The paper also establishes that action-space completeness (not just model scale) is a binding constraint on agent system performance, a key design consideration for harness architects.