Skip to content

The Complexity Trap: Simple Observation Masking Is as Efficient as LLM Summarization for Agent Context Management

🕒 Published (v1): 2025-08-29 09:02 UTC · Source: Arxiv · Venue: NEURIPS 25 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Simple observation masking — replacing old tool-output tokens with a placeholder — matches or beats LLM-based trajectory summarization on both cost and solve rate across five diverse model configurations on SWE-bench Verified. The added complexity of LLM summarization yields no consistent benefit over this O(1)-overhead baseline. A novel hybrid of both strategies squeezes out a further 7–11% cost reduction.

Problem

SE agents running on iterative agentic loops accumulate long, verbose context histories dominated by environment observations (~84% of tokens per turn). State-of-the-art scaffolds (OpenHands, Cursor) default to LLM-based trajectory summarization to manage this, but whether that added complexity outperforms simply dropping old observations has never been systematically evaluated.

Method

The paper formalizes two context management strategies within the SWE-agent scaffold and evaluates them on SWE-bench Verified (500 instances):

  • Observation Masking: a rolling-window function \(f_{\text{mask}}(\tau_{t-1}, M)\) that replaces environment observations \(o_i\) for \(i < t - M\) with a short placeholder string, preserving the full reasoning and action chain.
  • LLM-Summary: a separate summarizer LLM \(\pi'\) condenses turns \(T_{t_{\text{last}}+1}, \ldots, T_{t-1-M}\) into a running summary \(s_t \sim \pi'(\cdot | o_{si}, T_{\text{sum}})\), bounding context growth recursively.
  • Hybrid: applies Observation Masking first, then LLM-Summary on top.

Experiments span Qwen3-32B, Qwen3-Coder 480B, and Gemini 2.5 Flash (both thinking and non-thinking modes), with turn limit 250, \(M=10\) for both strategies, and \(N=21\) for LLM-Summary.

Key Contributions

  • Systematic, controlled comparison of Observation Masking vs. LLM-Summary across five model configurations on SWE-bench Verified.
  • Identification of a "trajectory elongation" side-effect: LLM-Summary increases mean trajectory length by ~13–15% compared to Observation Masking, partially explaining its lower cost efficiency.
  • Cost attribution showing that LLM summarization API calls themselves account for 0.65–7.2% of total instance cost, with limited cache reuse.
  • Novel hybrid approach (Observation Masking + LLM-Summary) reducing cost by 7% vs. Observation Masking alone and 11% vs. LLM-Summary alone on Qwen3-Coder 480B.
  • Initial generalization evidence on OpenHands (with scaffold-specific hyperparameter tuning of \(M\)).

Results

  • Cost reduction vs. Raw Agent: both strategies cut cost by >50% in most configurations (e.g., Gemini 2.5 Flash: Observation Masking −56.1%, LLM-Summary −41.5%).
  • Solve rate parity: Observation Masking matches or exceeds LLM-Summary in 4/5 configurations.
  • Qwen3-Coder 480B: Observation Masking 54.8% vs. LLM-Summary 53.8%; cost $0.61 vs. $0.64.
  • Gemini 2.5 Flash (thinking): Observation Masking 36.4% vs. LLM-Summary 31.4% (−5 pp for summarization).
  • Gemini 2.5 Flash (non-thinking): LLM-Summary marginally better (36.0% vs. 35.6%) but costlier ($0.24 vs. $0.18).
  • Trajectory elongation: LLM-Summary increases mean trajectory length ~15% over Observation Masking for Qwen3-Coder 480B and Gemini 2.5 Flash.
  • Hybrid on SWE-bench Verified-50 (Qwen3-Coder 480B): −7% cost vs. Observation Masking, −11% vs. LLM-Summary.

Limitations

  • SWE-bench Verified experiments use only 500 instances; hybrid evaluated on 50-instance slice due to cost.
  • OpenHands generalization tested only on a 50-instance slice with one model (Gemini 2.5 Flash non-thinking).
  • Optimal \(M\) is scaffold-specific and requires tuning; no automated hyperparameter selection method is proposed.
  • Thinking-mode results for Gemini 2.5 Flash show both strategies significantly hurt solve rate vs. Raw Agent (Observation Masking −9.9%†, LLM-Summary −22.3%†), suggesting context management interacts poorly with long reasoning chains.
  • Trajectory elongation mechanism is observed but not causally explained; the "reinforcing signal" hypothesis is not experimentally isolated.
  • No evaluation beyond SE agents (though authors note Tang et al.'s results suggest applicability to deep research and CUA).

Relevance to Harnesses / Meta-Harnesses

This paper directly informs the design of meta-harnesses that orchestrate long-running agentic loops: the finding that a stateless, zero-LLM-cost observation masking window matches expensive LLM summarization pipelines means harnesses need not implement a secondary summarizer model as a sub-agent. The "trajectory elongation" effect is a concrete failure mode for harness designers — LLM-Summary may cause agents to loop longer, increasing orchestration cost in ways that offset the compression benefit. The scaffold-specific hyperparameter sensitivity of \(M\) highlights that meta-harnesses wrapping multiple agent backends (e.g., SWE-agent vs. OpenHands) need per-backend context management tuning rather than a single universal policy. The hybrid strategy (masking + summarization) offers a composable primitive that harness builders can layer.