Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0¶
🕒 Published (v1): 2026-07-15 16:36 UTC · Source: Arxiv · link
Ask a follow-up
Open an assistant pre-loaded with this paper's context.
💬 Ask ChatGPT✦ Ask Claude
TL;DR¶
Most agent-optimizer evaluations are one-shot: optimize once on a fixed benchmark, report the gain. This paper asks whether those gains compound—whether a second round of optimization, on expanded tasks, builds on the first round without eroding it. Using a two-phase continual-learning protocol on Terminal-Bench 2.0 hard tasks, it finds that only RELAI-VCL (which enforces a no-regression constraint inside the search loop) achieves both positive transfer and continued improvement, reaching a lifelong average pass rate of 76.4% vs. 66.0% (GEPA) and 64.6% (Meta Harness).
Problem¶
Published agent-harness optimizers are evaluated in a static, single-phase setting: optimize on task set \(T_1\), measure improvement on \(T_1\). This conflates three distinct properties—static optimization strength, transfer to unseen tasks, and the ability to keep improving when new tasks arrive—and gives no signal about whether gains generalize or compound under repeated, recursive optimization as deployed agents encounter new failures over time.
Method¶
The paper introduces a two-phase phased continual-learning protocol on Terminal-Bench 2.0 hard tasks:
- Phase 1: Each optimizer receives 200 rollouts to optimize a common baseline agent (GPT-5.5 on the Harbor framework) against \(T_1\) (12 hard tasks, 900s timeout), producing \(A_1\).
- Transfer eval: \(A_1\) is evaluated—without further optimization—on \(T_1 \cup T_2\) (22 tasks total), measuring generalization to 10 unseen tasks \(T_2\).
- Phase 2: Each optimizer receives another 200 rollouts starting from its own \(A_1\) to optimize on \(T_1 \cup T_2\), producing \(A_2\), evaluated on the full union.
Metrics are Pass Rate at each stage plus: $\(\text{LifelongAvg} = \tfrac{1}{3}(\text{Phase-1} + \text{Transfer} + \text{Final})\)$
Three optimizers are compared under identical budgets: GEPA (reflective evolutionary prompt mutation), Meta Harness (agentic LLM proposer that edits harness code directly, using Codex/GPT-5.5), and RELAI-VCL (regression-aware continual-learning optimizer that rejects any candidate harness edit regressing on previously solved tasks—enforced in-loop rather than post-hoc).
Key Contributions¶
- A phased continual-learning evaluation protocol (Phase 1 → Transfer → Phase 2 re-opt) that decouples static optimization strength, transfer, and compoundability.
- A controlled head-to-head comparison of GEPA, Meta Harness, and RELAI-VCL under identical rollout budgets on Terminal-Bench 2.0.
- Empirical evidence that in-loop regression control is the decisive property separating compounding optimizers from non-compounding ones.
- Qualitative analysis linking GEPA's negative transfer to per-task hardcoded prompt content (prompt grew from 5 to 103 lines with named per-task lessons), and Meta Harness's stall under re-optimization to its conservative, generic changes.
- Discussion of what a more realistic continual-learning agent benchmark would require (correlated tasks, production-like failure distributions).
Results¶
| Agent | Phase 1 | Transfer | Final | Lifelong Avg |
|---|---|---|---|---|
| Baseline | 62.5% | 56.8% | 56.8% | 58.7% |
| GEPA | 70.8% | 54.5% | 72.7% | 66.0% |
| Meta Harness | 66.6% | 68.2% | 59.1% | 64.6% |
| RELAI-VCL | 79.2% | 72.7% | 77.3% | 76.4% |
- RELAI-VCL is the only method with positive transfer (+15.9 pp over baseline on unseen \(T_2\) tasks alone: 65.0% vs. ~50% baseline on \(T_2\)) and continued Phase 2 improvement.
- GEPA transfers below the unoptimized baseline (54.5% vs. 56.8%), consistent with prompt overfitting (\(T_1\)-specific lessons hardcoded into the prompt).
- Meta Harness transfers well (68.2%) due to generic harness robustness fixes, but stalls or regresses in Phase 2 (59.1%), suggesting its changes are not a suitable starting point for further optimization.
- RELAI-VCL leads at all four evaluation stages.
Limitations¶
- Small task set: \(|T_1| = 12\), \(|T_2| = 10\); results may not be statistically robust given only two trials per task.
- Terminal-Bench 2.0 tasks are largely independent of one another; real production failures are often correlated across a shared application domain.
- GEPA's code-mutation variant failed to produce a valid candidate in Phase 1 and was excluded from Phase 2, leaving only the prompt-only GEPA variant in the full comparison.
- Only one proposer model (GPT-5.5) is used across all methods; results may not generalize to other LLMs.
- The lifelong average is an unweighted mean of only three snapshots, not a continuous integral over the task stream.
- RELAI-VCL is a proprietary method from the same institution that designed the protocol, which introduces potential evaluation bias despite efforts to use identical budgets and a shared baseline.
Relevance to Agentic AI / LLM Agents¶
This paper directly challenges the dominant evaluation practice in agentic AI—reporting one-shot benchmark gains as if they were stable properties of an optimizer—and replaces it with a continual-learning lens that matches how optimization actually occurs in deployed systems. The finding that regression control must be inside the search loop, not applied post-hoc, is a concrete architectural prescription for anyone building self-improving agent pipelines. It also connects agentic harness optimization to catastrophic forgetting in continual learning, showing that the same structural instability that plagues neural network weight updates reappears at the level of prompt/code harness search. For researchers tracking LLM agents, this establishes a new evaluation dimension—compoundability—that should sit alongside raw benchmark performance when assessing any agent-optimization method.