Skip to content

Is Progressive Disclosure All You Need for Long-Context Agents?

🕒 Published (v1): 2026-07-20 06:35 UTC · Source: Arxiv · link

Why this paper was selected

Benchmarks progressive-disclosure vs retrieval for long-context agents; direct harness architecture guidance

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper runs the first controlled experiment testing whether Agent Skills–style progressive disclosure (a practitioner pattern for packaging long documents as on-demand skill folders) improves long-document QA over raw-document navigation or classical retrieval. On a single book, gains depend entirely on the agent harness; at library scale (20 books), flat one-level disclosure pulls decisively ahead while hierarchical multi-level disclosure never helps and sometimes collapses accuracy.

Problem

Practitioners have adopted the Agent Skills progressive-disclosure pattern — structuring a long document as a SKILL.md-rooted folder that an agent reads on demand — as an alternative to loading full documents into context or using external retrievers. This adoption was driven by engineering intuition with no controlled empirical validation: no study had compared raw-document navigation, flat disclosure, and hierarchical disclosure against each other and against a retrieval baseline on a standardized long-document benchmark.

Method

The authors define three approaches sharing a fixed chunk set, isolating only the routing wiring: - raw: the book is placed on a sandboxed filesystem; the agent navigates it freely with no imposed structure. - flat: a single SKILL.md with an always-loaded description routes the agent to an in-body table indexing chunk files under references/; chunks load on demand. - hierarchical: each chunk becomes its own skill with an always-loaded description; a meta-router at the top routes among them; routing recurses one level deeper and per-chunk descriptions pay always-loaded context cost.

Both skill packs are built via the book-to-skill pipeline: split on chapter headings (or fixed ~4000-word paragraph-boundary chunks as fallback), attach LLM-written chunk descriptions and one book-level description.

The benchmark is LOONG DOC, which converts ∞Bench book-QA items into interactive BenchFlow tasks: the agent answers via filesystem tool calls (read/grep/open) inside a sandbox, with a per-trajectory scalar reward from a deterministic verifier. Three harnesses (Codex, Pi, Claude-Code) × three model families (gpt-5.4-mini, qwen3.6-27b, claude-haiku-4.5) × single-book and library-scale (\(K \in \{5, 10, 20\}\) books) are evaluated, with hybrid-rag (retrieve-and-rerank) as an external reference.

Key Contributions

  • Controlled three-way comparison of raw, flat, and hierarchical agentic reading strategies over a fixed chunk set (only the routing wiring varies).
  • LOONG DOC: a reproducible interactive long-context agent environment built on BenchFlow and ACP, logging full agent trajectories with per-run rewards suitable for post-training.
  • First systematic empirical study of the progressive-disclosure pattern on a standardized book-QA benchmark (∞Bench) across harnesses, models, and corpus scales.

Results

Single-book (Table 1): - Flat disclosure matches or exceeds raw on Pi and Claude-Code in nearly every cell (e.g., Pi gpt-5.4-mini En.MC: raw=0.8851, flat=0.9126). - Codex ties across all three methods: raw navigation and flat/hierarchical are indistinguishable within error on all subsets, because Codex's bare agent already greps for named entities and reads matched passages (self-constructed retrieve-then-read). - Hierarchical is never better than flat and is actively harmful on Pi: En.MC collapses from 0.9126 (flat) to 0.6398 (hierarchical) on gpt-5.4-mini; Zh.QA drops from 0.7479 to 0.3890 on qwen3.6-27b. Always-loaded child descriptions saturate the router's context before it commits to a chunk. - hybrid-rag trails both raw and flat in most cells (e.g., Pi gpt-5.4-mini En.QA: flat=0.7259, hybrid-rag=0.6214).

Library scale (\(K=20\), Table 2 + Fig. 4): - Raw navigation collapses for the harness that matched flat single-book: Codex En.QA falls from ~0.657 (\(K=5\)) toward 0.257 (\(K=20\)) under raw, while flat holds at 0.462 (\(K=20\)). - Corpus scaling is a strict efficiency loss: accuracy falls and cost-per-fixed-accuracy rises simultaneously as \(K\) grows. - The cost–accuracy Pareto frontier is dominated by Codex·raw at \(K=5\) but shifts to Codex·flat at \(K \geq 10\). - Hierarchical does not rescue scaling; it matches or trails flat at library scale as well.

Limitations

  • Evaluation is limited to ∞Bench (English and Chinese book QA); generalization to other long-document domains (legal, scientific, code) is untested.
  • The book-to-skill pipeline uses LLM-written chunk descriptions whose quality affects routing metadata; only the fixed-structure variant is tested (not fully model-authored skills, which prior work found null).
  • The multi-book scaling study uses a limited sample of bundles and sizes (\(K \leq 20\)); the crossover point and degradation curve at larger \(K\) are not characterized.
  • Three harnesses and three model families cover a range but not the full spectrum; results are explicitly harness-dependent, limiting universal prescription.
  • The RL/post-training potential of LOONG DOC's per-trajectory rewards is proposed but not demonstrated.

Relevance to Agentic AI / LLM Agents

This paper directly empirically tests the Agent Skills / progressive-disclosure design pattern that has become a de-facto standard in agentic engineering (e.g., Claude Code's own architecture), providing the first controlled evidence for when and why it helps. The central finding — that disclosure buys context (navigation bandwidth), not intelligence, and is redundant when a strong agent harness already self-assembles locate-then-read behavior — is a sharp constraint on when skill packs earn their design cost. For researchers building or evaluating long-context agents, the flat-vs-hierarchical ablation warns that deeper routing hierarchies reliably hurt due to always-loaded context saturation, making one-level flat indexing the empirically defensible default. The LOONG DOC environment (ACP-compatible, trajectory-logging, with per-run rewards) is a reusable infrastructure contribution for the agent evaluation and training community.