A Self-Evolving Framework for Efficient Terminal Agents via Observational Context Compression

🕒 Published (v1): 2026-04-21 15:25 UTC · Source: Arxiv · link

Why this paper was selected

Observational context compression for efficient long-horizon terminal agent harnesses

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TACO is a plug-and-play, unsupervised terminal-observation compression adapter for CLI agents that autonomously discovers, refines, and reuses structured compression rules from interaction trajectories. Rather than summarizing or truncating naively, it performs preservation-aware filtering — suppressing redundant terminal noise while keeping exact error messages, file paths, and test names intact. Across six benchmarks it reduces token consumption by 12%–27% while maintaining or improving task-success rates.

Problem

Long-horizon CLI agents accumulate raw terminal output (build logs, install traces, test reports) that is heterogeneous and low-information-density: manual inspection of 50 TB 2.0 trajectories shows 24.6%–44.1% of prompt tokens are removable redundancy. Generic LLM summarization may paraphrase or omit exact strings critical for downstream actions; static heuristics are brittle across command types and repositories; training-based pruners (e.g. SWE-Pruner) require labeled data and are benchmark-specific. No existing method autonomously learns preservation-aware compression rules from agent trajectories without human engineering.

Method

TACO wraps any terminal agent as a plug-in adapter with three components:

1. Terminal Observation Compression. At each step the raw observation \(O_t\) is classified as Critical (explicit error/exception traces) or non-Critical. Critical outputs pass through unchanged; non-Critical outputs are filtered by the active task rule set \(R_t\) via a conservative rule executor \(F_{R_t}\): $\(\tilde{O}_t = \begin{cases} O_t & \text{if } O_t \text{ is Critical} \\ F_{R_t}(O_t \mid C_t) & \text{otherwise} \end{cases}\)$

2. Intra-Task Rule Evolution. When an observation matches no rule, an LLM proposes a new rule added to \(R_t\). Over-compression is detected via implicit agent feedback (requests for full output, repeated commands); implicated rules are suppressed and replaced with more conservative LLM-generated variants.

3. Global Rule Pool (GRP). Rules are shared across tasks. Each rule \(r\) carries a global confidence \(c^g_r\) and cumulative successful-application count \(n_r\); its ranking score is: $\(R^g_s(r) = c^g_r \cdot (n_r + 1)\)$ Complaints decay \(c^g_r\); successful use increments \(n_r\). Before each task, top-\(k\) rules are retrieved and an LLM performs task-conditioned selection, filtering globally high-ranked but locally irrelevant rules. After each task, only rules with \(\Delta n_r \ge 1\) and task confidence \(c^t_r \ge \tau\) are written back.

Convergence is assessed by a reward-free Retention metric — the fraction of top-\(K\) rules that remain stable across consecutive evolution rounds — without accessing any benchmark labels or verifier outcomes.

Key Contributions

• First self-evolving, unsupervised, plug-and-play terminal-observation compression framework (TACO) requiring no human-crafted rules, task-specific training, or reward signals.
• Global Rule Pool mechanism that accumulates, scores, and transfers compression knowledge across heterogeneous tasks.
• Reward-free convergence criterion (Retention) that detects rule-frontier stability from observation-level signals alone.
• Empirical demonstration that preservation-aware compression outperforms both maximal compression (LLM summarization) and static curated rules.

Results

• TerminalBench 1.0 & 2.0 (main): +1%–4% absolute accuracy gains over Terminus-2 baseline across six backbone models; under matched token budgets, +2%–3% accuracy improvement.
• Cross-benchmark token reduction (MiniMax-M2.5 backbone):
• SWE-Bench Lite: 307.61M → 270.53M tokens (↓12.1%), accuracy 56.30 → 57.12 (+0.82)
• CompileBench: ↓21.6% tokens, accuracy unchanged at 75.00
• DevEval: ↓27.0% tokens, accuracy 38.10 → 39.74 (+1.64)
• CRUST-Bench: ↓17.5% tokens, accuracy 47.00 → 48.05 (+1.05)
• TB 1.0: ↓21.2% tokens, accuracy 42.30 → 45.25 (+2.95)
• TB 2.0: ↓2.7% tokens, accuracy 42.80 → 44.16 (+1.36)
• Per-step token reduction (TB 2.0): ~8–12% for large models (Qwen3-Coder-480B, DeepSeek-V3.2); smaller models gain longer successful trajectories at cost of marginal per-step savings.
• vs. static baselines (Qwen3-Coder-480B, TB 2.0): TACO (25.9±1.5 acc, 10.78% tok. reduction) beats 200 human-curated rules (24.3±2.2, 17.90%), LLM summarization (20.3±3.2, 21.30%), and static LLM-generated rules (19.71±1.6, 7.10%).
• Compression quality (N=200 sampled observations): 96.0% preservation of critical information, 81.0% redundancy removal, 78.5% retention of useful non-critical context; only 4.0% flagged as potential critical loss.
• Auxiliary overhead: Rule-evolution LLM calls account for <2% of total token consumption across all benchmarks.

Limitations

• Rule discovery depends on the quality of the LLM used as the rule-proposer; weak backbone models may generate poor rules, potentially harming smaller-model scenarios.
• Token reduction on TB 2.0 is only 2.7%, suggesting that very short or highly compressed observations limit headroom in some settings.
• Batch size \(N\) affects rule propagation speed and final performance (analyzed only in appendix); optimal \(N\) is not analytically determined.
• Evaluation uses fixed agent scaffolds (Terminus-2, Mini-SWE-Agent); interaction with other scaffolds is not systematically studied.
• The convergence criterion (Retention) is validated only on three models on TB 2.0; generality to other domains is assumed, not demonstrated.

Relevance to Harnesses / Meta-Harnesses

TACO is structurally a meta-harness layer: it wraps arbitrary agent scaffolds without modifying them, intercepting observations between environment and agent to apply dynamically evolving compression policies. The Global Rule Pool is an explicit cross-task knowledge accumulation mechanism — a canonical meta-harness concern — enabling compression knowledge discovered in one workflow to transfer to subsequent tasks without retraining the base agent. The self-evolution loop (propose → execute → get implicit feedback → update pool → re-initialize next task) mirrors the self-improving harness paradigm where the orchestration layer refines its own behavior from trajectory signals. For researchers building meta-harnesses, TACO demonstrates that observation preprocessing (not just action generation) is a productive axis for autonomous harness self-improvement, and that reward-free convergence signals derived from environment observations can substitute for expensive labeled feedback.