Solving Context Window Overflow in AI Agents

🕒 Published (v1): 2025-11-27 19:22 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LLM agents fail when tool outputs exceed the context window; truncation and summarization lose information. This paper introduces a pointer-based indirection layer that stores large tool outputs in runtime memory and exposes only short path identifiers to the LLM, enabling arbitrarily large tool responses without information loss or architectural changes.

Problem

Tool-calling LLM agents are blocked when a single tool response exceeds the model's context window — a routine occurrence in domains like Materials Science and Chemistry (e.g., a \((128, 128, 128)\) float32 electronic grid = 2,097,152 elements). Existing mitigations (truncation, summarization, selective tool inclusion) all discard information and are therefore inapplicable to pipelines that must pass complete tool outputs as inputs to downstream tools.

Method

Each tool in the agent's toolkit is wrapped with a mirrored tool consisting of three components: (1) an input inspector that detects whether an argument is a raw value or a memory path pointer, (2) the original unmodified tool, and (3) a post-processing module that writes outputs exceeding the context window to a keyed runtime memory store and returns a hierarchical path string (e.g., generate_molecule_grid-<uuid>/raw_grid) instead of the raw data. The LLM receives and propagates only these short path strings; mirrored wrappers resolve them back to values before invoking the underlying tool. A dedicated retrieve_final_answer_from_memory tool surfaces results to the user. No changes to the LLM architecture or the original tools are required; the method is compatible with MCP servers and any ReAct-style agentic framework.

Key Contributions

• Pointer-based indirection that decouples LLM context from tool output size, enabling processing of arbitrarily large responses with zero information loss.
• Mirrored tool wrapper design that is transparent to the agent and requires no modification to underlying tools or the LLM.
• Demonstrated ~7× token reduction even on workflows where outputs fit within the context window (SDS extraction: 6,411 → 842 tokens; 43 → 11 seconds).
• Enables previously infeasible agentic pipelines (molecule grid retrieval collapses conventional workflow at ~20.8M tokens; pointer method completes in ~1,234 tokens).

Results

• Molecule grid retrieval (Experiment 3.1): Conventional workflow fails outright (context overflow at estimated ~20,822,181 tokens average across 50 runs). Proposed method completes in avg. 1,234 tokens and 33.87 seconds (50 runs).
• SDS ingredient extraction (Experiment 3.2): Conventional workflow: avg. 6,411 tokens, 43.05 seconds. Proposed method: avg. 842 tokens, 11.05 seconds — approximately 7.6× fewer tokens, 3.9× faster.
• Both experiments used Llama 4 Maverick 17B-128E-Instruct with the BeeAI framework under a ReAct pattern.

Limitations

• Runtime memory is ephemeral; persistence across sessions or agents is not addressed.
• No mechanism for the agent to selectively read or transform a sub-slice of a memory element within context constraints (acknowledged as future work).
• Evaluation is limited to two single-domain use cases; no systematic benchmark across diverse tool types or models.
• The paper does not evaluate failure modes when memory path resolution itself fails or when hierarchical key schemas are malformed.
• Token counts are estimates for the conventional-approach overflow case, not direct measurements.

Relevance to Harnesses / Meta-Harnesses

Pointer-based memory indirection is directly applicable to meta-harness design: a harness orchestrating multiple tool-calling agents must manage data flow between stages, and large intermediate artifacts (embeddings, parsed documents, simulation outputs) are the primary bottleneck preventing pipeline composition. This work provides a clean, framework-agnostic pattern — mirrored tool wrappers + keyed runtime memory — that a meta-harness can adopt as a standard middleware layer, decoupling orchestration logic from data size. It also establishes a concrete token-efficiency argument for why harnesses should prefer pointer passing over value passing between agent steps, directly informing design choices in multi-step digest and analysis pipelines.