Skip to content

CompactionRL: Reinforcement Learning with Context Compaction for Long-Horizon Agents

🕒 Published (v1): 2026-07-06 17:55 UTC · Source: Arxiv · link

Why this paper was selected

CompactionRL solves finite-context failure in long-horizon agents; Jie Tang + Yuxiao Dong (THU)

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CompactionRL is a PPO-based RL framework that integrates context compaction—periodic summarization of interaction history—directly into training for long-horizon LLM agents, enabling effective task execution beyond the model's peak context window. By jointly optimizing execution and summarization under a shared task reward, with token-level loss normalization and cross-trajectory GAE, it consistently outperforms base models and standard PPO with inference-only compaction on agentic coding benchmarks.

Problem

Long-horizon agentic tasks (e.g., software engineering, terminal interaction) generate trajectories that frequently exceed LLM context windows before task completion. Existing RL pipelines (e.g., GRPO, standard PPO) assume fixed-context rollouts and cannot accommodate compaction, where a rollout is split into a variable number of segments with different lengths and shared terminal rewards—creating length-bias in loss weighting and incorrect temporal credit assignment across segment boundaries.

Method

CompactionRL wraps PPO with trainable in-rollout context compaction. During rollout collection, when remaining context budget \(C - |h_t| < T_{\text{comp}}\), the policy generates a summary \(S_t \sim \pi_\theta(\cdot \mid h_t \oplus q_{\text{sum}})\), and execution resumes from a reconstructed context \(\bar{h}_t = (s) \oplus u_{\text{resume}}(S_t) \oplus (z_{t-k+1}, \ldots, z_t)\) retaining the \(k=2\) most recent steps. Both execution and summarization token sequences are included in the RL objective under the same final task reward.

Two key training fixes handle the resulting variable-length multi-segment structure:

  1. Token-level loss normalization: The PPO objective is normalized over the total count of optimized assistant tokens \(|M|\) rather than over segments, removing bias from variable segment counts and lengths.

  2. Cross-trajectory GAE: The local per-segment advantage \(A^{\text{loc}}_{s,i}\) is corrected by the number of optimized tokens \(N_{>s}\) in subsequent segments: \(\hat{A}_{s,i} = (\gamma\lambda)^{N_{>s}} A^{\text{loc}}_{s,i}\), so that discount matches each token's true temporal distance to the terminal reward in the concatenated rollout.

Training uses the SWE-Dev dataset with a global batch size of 128, group size of 1, context budgets of 64k (30B) and 80k (106B), value pretraining for 50 steps, and length-adaptive GAE (\(\lambda = 1 - \frac{1}{\alpha l}\), \(\alpha = 1.5\)).

Key Contributions

  • First RL framework to incorporate trainable context compaction into long-horizon agentic LLM training, extending the effective training horizon beyond the peak working context length.
  • PPO formulation replacing group-wise methods (GRPO) to handle variable compaction-induced segment counts without fixed-group reward normalization.
  • Token-level loss normalization eliminating segment-count and segment-length bias across compacted rollouts.
  • Cross-trajectory GAE preserving correct temporal credit assignment across compaction boundaries.
  • Empirical demonstration that jointly optimizing summarization via RL (vs. treating it as an inference-time heuristic) provides substantial task-level gains.
  • Deployment of CompactionRL in the production RL pipeline for GLM-5.2 (750B-A40B).

Results

  • GLM-4.7-Flash (30B-A3B) at 64k peak length, compacted (\(\times4\)) evaluation:
  • SWE-bench Verified: 56.0% (base: 50.5%; +5.5 pts over base; +8.0 over standard RL without compaction)
  • Terminal-Bench 2.0: 20.2% (base: 13.4%; +6.8 pts over base)
  • GLM-4.5-Air (106B-A30B) at 80k peak length, compacted (\(\times4\)) evaluation:
  • SWE-bench Verified: 66.8% (base: 59.8%; +7.0 pts over base; +4.3 over standard RL)
  • Terminal-Bench 2.0: 24.5% (base: 21.4%; +3.1 pts over base)
  • Standard PPO without compaction improves single-window performance but does not transfer consistently to compacted inference; CompactionRL is the only variant with consistent gains in the compacted setting.
  • Ablation: Summary agent quality alone accounts for up to 6.5 absolute points on SWE-bench Verified (Table 1), motivating trained rather than fixed summarizers.
  • Ablation: Removing token-level loss causes a larger drop than removing cross-trajectory GAE, suggesting length-bias correction is the more critical component (Table 4).
  • CompactionRL at 64k peak often matches or exceeds standard RL trained at 128k (Long), achieving comparable effective horizon at half the context budget.

Limitations

  • CompactionRL does not consistently improve single-window (non-compacted) performance; it trades off single-window gains for compacted-setting gains due to training distribution mismatch.
  • Evaluation on SWE-bench Verified uses a random 200-instance subset rather than the full benchmark, making direct comparison with public leaderboard baselines approximate.
  • The approach requires a critic model (same scale as the policy), increasing compute relative to critic-free methods like GRPO.
  • No separate summary-quality reward is used; summarization quality is shaped solely through task-level outcome reward, which may be a weak signal for summary learning.
  • Experiments are limited to agentic coding tasks; generalization to other long-horizon domains (web interaction, tool use) is not demonstrated.
  • At most three compaction operations per rollout are allowed; behavior under more frequent or deeper compaction chains is not studied.

Relevance to Agentic AI / LLM Agents

CompactionRL directly addresses one of the central bottlenecks in deploying LLM agents on real-world long-horizon tasks: finite context. By making context compaction a trained policy component rather than an inference-time heuristic, it shifts the framing from "how to fit trajectories into a window" to "how to learn to compress and resume effectively"—a fundamentally agentic skill. The cross-trajectory GAE and token-level loss contributions are generalizable training-infrastructure solutions applicable to any multi-segment agentic RL setup, not just compaction. Its deployment in GLM-5.2 training signals that compaction-aware RL is becoming a practical ingredient in frontier agent pipelines, relevant to anyone building or evaluating long-horizon coding, reasoning, or interactive agents.