Skip to content

Is Progressive Disclosure All You Need for Long-Context Agents?

🕒 Published (v1): 2026-07-20 06:35 UTC · Source: Arxiv · link

Why this paper was selected

Progressive disclosure as context management primitive for long-context agents; empirical comparison vs retriever

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper presents the first controlled evaluation of the "Agent Skills" progressive-disclosure pattern — wrapping a long document in a SKILL.md-rooted folder structure so an agent loads only what a query needs — against raw-document navigation and classical hybrid-RAG across three agent harnesses and three model families on ∞Bench. The main finding is that disclosure buys context, not intelligence: it is redundant when the agent harness already navigates well (Codex grepping raw text), decisive when the corpus spans many books, and one flat level is always enough because a second routing level saturates context and collapses accuracy.

Problem

Long-document QA has been handled by either full-context loading or external retrieval, but practitioners rapidly adopted Agent Skills (SKILL.md-based progressive disclosure) as a third path on engineering intuition — with no controlled comparison against raw navigation or retrieval on a standardized benchmark, and without isolating which design axis (routing depth vs. index placement) drives any benefit.

Method

The authors implement three agentic reading conditions over a shared chunk set built once per book via the "book-to-skill" pipeline (chapter-boundary splitting → LLM-written per-chunk descriptions → book-level description):

  • raw: book files on disk, no skill pack, harness navigates freely.
  • flat: single SKILL.md whose always-loaded description gates the book; body contains a chunk index table; chunks live under references/ and load on demand.
  • hierarchical: meta-router SKILL.md plus per-chunk child skills, so chunk descriptions are always-loaded (paying constant context cost) and chunk content loads on activation.

They wrap ∞Bench into LOONGDOC, an interactive BenchFlow environment that stages books + skill packs in a sandboxed file system, runs any ACP-compatible harness, records full trajectories (tool calls, files opened, tokens), and emits per-trajectory scalar rewards. Corpus scale is tested at \(K \in \{1, 5, 10, 20\}\) books per task. Baselines include hybrid-RAG (retrieve-and-rerank). Three harnesses (Codex, Pi, Claude-Code) × three models (gpt-5.4-mini, qwen3.6-27b, claude-haiku-4.5) × three seeds.

Key Contributions

  • Controlled, within-chunk-set comparison of raw / flat / hierarchical agentic reading, isolating routing depth and index placement as independent variables.
  • LOONGDOC: an interactive, trajectory-logging long-context agent environment built on BenchFlow, converting ∞Bench items to ACP tasks with per-run reward signals (reusable for RL post-training).
  • First systematic empirical measurement of progressive disclosure on a standardized long-document benchmark, explaining gains via trajectory inspection rather than aggregate scores alone.

Results

Single-book (K=1): - Flat beats raw on Pi and Claude-Code in nearly every cell (e.g., Pi gpt-5.4-mini En.MC: raw 0.8851 → flat 0.9126). - Codex ties flat on raw for all subsets; trajectory inspection shows Codex's bare agent already greps entity names and reads only matched passages — making disclosure redundant. - Hierarchical never beats flat; actively harmful on Pi: En.MC drops from 0.9126 (flat) to 0.6398 (hierarchical) on gpt-5.4-mini, and Zh.QA drops from 0.7479 to 0.3890 on qwen3.6-27b — always-loaded child descriptions saturate the router context. - hybrid-RAG trails both raw and flat on Codex in nearly every cell.

Library scale (K=5–20): - Raw collapses as \(K\) grows: En.QA falls to 0.26 at K=20 vs. flat at 0.46 (gap clears one standard error). - Flat degrades more slowly than raw; becomes the Pareto-dominant condition at \(K \geq 10\) on the cost–accuracy frontier. - Hierarchical does not reproduce the scaling rescue that flat achieves. - Larger corpus is strictly less efficient: accuracy falls and cost-per-fixed-accuracy rises simultaneously.

Limitations

  • ∞Bench is English/Chinese book QA; generalization to other document types (legal, scientific, code corpora) is untested.
  • Paper text is truncated at K=5/10/20 tables; some cells are omitted (En.QA K=10/20 for several conditions).
  • Chunk granularity and LLM-written description quality are fixed; interaction with these variables is not fully isolated.
  • LOONGDOC token costs are reported as uncached upper-bound estimates, not real production costs.
  • The post-training use case (using per-trajectory reward for RL) is proposed but not demonstrated.
  • Only three harnesses tested; findings about harness-dependence may not generalize to other ACP-compatible harness designs.

Relevance to Harnesses / Meta-Harnesses

This paper is directly about how harness architecture determines whether a meta-structure (skill packs / progressive disclosure) adds value: the same SKILL.md artifact is redundant under Codex (which internally reconstructs locate-then-read) and decisive under Pi and Claude-Code, making the harness the confounding variable that prior advocacy of Agent Skills ignored. LOONGDOC is itself a meta-harness contribution — a BenchFlow-based environment that wraps any ACP-compatible harness with sandboxed file systems, staged skill packs, and trajectory logging, establishing a reproducible testbed for comparing harness behaviors on long-context tasks. The finding that two routing levels break accuracy (by saturating always-loaded context) is a concrete architectural constraint for anyone designing hierarchical SKILL.md-based meta-harnesses.