Skip to content

Scaling External Knowledge Input Beyond Context Windows of LLMs via Multi-Agent Collaboration

🕒 Published (v1): 2025-05-27 17:45 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ExtAgents is a multi-agent framework that enables LLMs to consistently improve task performance by scaling external knowledge input beyond their context windows without retraining. It addresses two bottlenecks in existing multi-agent orchestration—limited synchronization bandwidth and reasoning-context overload—via global knowledge synchronization and knowledge-accumulating reasoning. On multi-hop QA benchmarks, ExtAgents outperforms state-of-the-art non-training methods at all input scales from 8k to 1024k tokens.

Problem

Existing context-window extension strategies (RAG, compression, multi-agent MapReduce-style methods) fail to reliably improve performance as external knowledge input grows beyond the LLM's context window. Specifically, LLM×MapReduce degrades relative to direct truncation when input exceeds 128k tokens. Two shared design bottlenecks explain this: (i) local synchronization bandwidth between agents is too narrow (bandwidth = 2 for Chain of Agents / LongAgent; \(O(L/|m|)\) for MapReduce), causing salient information to be lost across the agent graph; and (ii) the reasoning agent receives a flat dump of all messages, creating redundant-information overload that hurts answer quality.

Method

ExtAgents partitions the full knowledge source \(K = \{d_1, \ldots, d_N\}\) into fixed-size chunks (e.g., 8k–16k tokens) assigned to \(N\) Seeking Agents and one Reasoning Agent.

Global Knowledge Synchronization. At each timestep \(t\), every Seeking Agent reads the global message set \(M_{t-1} = \{m_{j,t-1}\}_{j=1}^N\) from the prior round. Because all-to-all exchange can overflow the window, each agent distributedly scores message relevance (\(h_{i,t} \in \mathbb{R}^+\)) and assembles only the top-\(k\) messages that fit: $\(m_{i,t}^{(\text{EA})} = a_{i,t}\!\left(q,\, d_i,\, \text{Top}_k(M_{t-1})\right), \quad |q|+|d_i|+|\text{Top}_k(M_{t-1})| < L.\)$ This gives full (global) bandwidth instead of a local neighborhood.

Knowledge-Accumulating Reasoning. After synchronization converges at timestep \(t^\star\), the Reasoning Agent iterates through doubling context windows: at iteration \(s\) it ingests the top-\(2^s\) messages, $\(M_r^{(s)} = \text{Top}_{2^s}\!\left(\{m_{i,t^\star}\}_{i=1}^N\right),\)$ checking answerability at each step and halting as soon as an answer is producible, avoiding information overload. If no answer is found after \(S\) iterations, a new synchronization round starts.

All Seeking Agents run in parallel per timestep; an interleaved async pipeline further overlaps reasoning and synchronization.

A new benchmark, ∞Bench+, filters samples from ∞Bench that are answerable within any single 8k-token chunk (using GPT-4o-mini as an oracle), leaving only genuine multi-hop queries that require cross-document aggregation (294 En.QA + 184 Zh.QA samples).

Key Contributions

  • Formal definition of the "scaling external knowledge input beyond context windows" problem, distinguishing it from scaling retrieval databases.
  • ∞Bench+: an unbiased multi-hop long-document QA benchmark constructed by filtering shortcut-solvable samples from ∞Bench.
  • Identification of the two bottlenecks (synchronization bandwidth, reasoning redundancy) shared across existing LLM-based context-extension multi-agent systems.
  • ExtAgents framework with global knowledge synchronization and knowledge-accumulating reasoning that overcomes both bottlenecks without additional training.
  • Demonstrated generalization across QA (HotpotQA, ∞Bench+) and long survey generation (AutoSurvey benchmark) with multiple LLM families.

Results

  • HotpotQA (F1), gpt-4o-mini: ExtAgents 0.534 (1024k input) vs. LLM×MapReduce 0.374 (128k), IterDRAG 0.413 (128k), Direct Input 0.204 (128k).
  • HotpotQA (F1), Llama-3.1-8B: ExtAgents 0.412 (1024k) vs. LLM×MapReduce 0.254 (256k), IterDRAG 0.368 (32k).
  • ∞Bench+ En.QA (F1), Llama-3.1-8B: ExtAgents 0.291 (1024k) vs. LLM×MapReduce 0.347 (256k) — competitive despite 4× more input.
  • ∞Bench+ Zh.QA (F1), gpt-4o-mini: ExtAgents 0.482 (1024k) vs. DRAG 0.436 (128k).
  • Long Survey Generation, gpt-4o-mini: ExtAgents LLM-as-a-Judge score 7.63 vs. AutoSurvey 6.75; 191 vs. 113 citations; lower duplication rate (1.80 vs. 2.41).
  • Performance monotonically increases with input length for ExtAgents across all tested scales; competing methods plateau or regress past 128k.
  • With gpt-4o-2024-08-06 on HotpotQA: ExtAgents (N=1 ablation) 0.553, full ExtAgents 0.597 at 1024k.

Limitations

  • Optimal chunk size is task-dependent and requires tuning per dataset.
  • The ∞Bench+ filtering leaves a small sample count, making evaluation variance non-negligible; the authors mitigate by including all ∞Bench samples >128k tokens.
  • Costs scale with the number of Seeking Agents (N); the paper measures latency but does not report total token cost vs. baselines at matched budgets.
  • Long-reasoning chain models (DeepSeek-R1-Distill-Llama-8B) do not benefit as much from additional context, suggesting the approach is less synergistic with chain-of-thought-heavy inference.
  • The framework is evaluated on QA and survey generation; generalization to other knowledge-intensive tasks (e.g., code retrieval, structured extraction) is not demonstrated.

Relevance to Harnesses / Meta-Harnesses

ExtAgents is a concrete multi-agent orchestration harness design: it defines a fixed two-role agent topology (Seeking + Reasoning), prescribes multi-round synchronization protocols with shared ranking signals, and implements a termination/escalation policy (answerability check → new sync round) — precisely the concerns of a meta-harness that coordinates heterogeneous agents over long-horizon tasks. The paper's systematic taxonomy of existing orchestration designs (Chain of Agents, LongAgent, MapReduce) as pipelines with quantifiable bottlenecks (bandwidth, redundancy ratio) offers a principled vocabulary for evaluating harness architectures. The knowledge-accumulating reasoning pattern — exponentially growing context windows with early-exit — is a generalizable harness primitive for any pipeline that must merge distributed partial results into a single reasoner. Benchmark construction (∞Bench+) as a harness-testing artifact is also noteworthy for anyone building evaluation scaffolds for multi-agent systems.