SEAGym: An Evaluation Environment for Self-Evolving LLM Agents¶

🕒 Published (v1): 2026-06-16 05:50 UTC · Source: Arxiv · link

Why this paper was selected

Defines and benchmarks 'agent harness' as execution layer; Tsinghua; fills key evaluation gap

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

SEAGym is an evaluation environment that converts static agent benchmarks into dynamic self-evolution task sources with train batches, frozen validation views, held-out ID/OOD transfer, replay diagnostics, and cost records. It formalizes the mutable agent harness — prompts, memory, tools, middleware, orchestration logic — as the evolving object, and provides a common rollout/update interface for comparing heterogeneous self-evolving methods. Experiments on Terminal-Bench 2.0 and HLE with ACE, TF-GRPO, and AHE reveal that validation gains routinely fail to transfer, useful intermediate snapshots can catastrophically regress, and batch size effects are non-monotonic.

Problem¶

Existing agent benchmarks evaluate a fixed agent in isolated episodes: state is reset between tasks, and the score reflects one frozen snapshot. This design removes the persistent harness state that self-evolving agents are meant to exploit, making it impossible to measure whether an update produces reusable improvement, overfits recent tasks, causes forgetting, or increases cost. Sequential/lifelong benchmarks partially address ordering but leave out single-task adaptation, epoch-based learning, forgetting diagnostics, and held-out transfer at different distribution distances — so gains reported by different methods are not comparable under a shared protocol.

Method¶

SEAGym models a self-evolution run as an MDP

\[M = (S, A, P, R, \rho)\]

where each state \(s_t\) holds the current agent snapshot, schedule position, and task context. An agent snapshot is

\[A_t = (M, H_t)\]

with fixed base model \(M\) and mutable harness state \(H_t\) (prompts, memories, skills, tools, middleware, runtime config). The environment samples a train batch \(B_t\), the agent produces trajectories \(T_t\) and receives feedback \(F_t\), then applies its own update rule

\[H_{t+1} = U(H_t, B_t, T_t, F_t).\]

SEAGym specifies only the environment side — task sampling, feedback visibility policy, schedule, and snapshot assessment — leaving \(U\) entirely to each method. Concretely:

Harbor integration: Harbor handles containerized task execution and verifiers; SEAGym wraps it to add train batches, frozen evaluation views, and snapshot timing without rewriting benchmarks.
Schedule parameters: state reset, task reuse, batch size, update timing, and epoch count are explicit variables, enabling single-task, online, and epoch-based settings under one protocol.
Data splits vs. evaluation views: base splits \(D_\text{train}/D_\text{val}/D_\text{test}\) control update evidence; evaluation views — update-validation \(V_\text{update-val}\), ID transfer \(V_\text{ID}\), OOD transfer \(V_\text{OOD}\), and replay \(V_\text{replay}\) — are frozen assessment lenses materialized from those splits.
Rollout/update interface: a thin wrapper converts SEAGym trajectory batches to a method's native update input and saves the resulting \(H_t\); this is the only integration requirement.
Metrics: Update Validation Gain \(\text{UVG}_i = \text{Perf}(E_i, V) - \text{Perf}(E_{i-1}, V)\), IDG/OODG (gain over initial snapshot on held-out views), Forgetting Rate \(\text{FR} = \max(0, \text{Perf}(A_0, D_R) - \text{Perf}(A_T, D_R))\), plus token/cost records.

Key Contributions¶

Formal MDP-style specification of self-evolution that separates environment (task sampling, feedback, schedule) from method (policy and harness update rule \(U\)).
SEAGym environment wrapping Harbor-compatible benchmarks into dynamic self-evolution task sources with configurable schedules.
Five complementary evaluation views: update-validation, ID transfer, OOD transfer, replay/retention, and cost — each exposing a different failure mode invisible to final-score-only evaluation.
Saved snapshot/metric artifacts and process diagnostics (harness diffs, trajectory references) for offline post-hoc analysis of why a run improves, regresses, or becomes expensive.
Empirical comparison of ACE, TF-GRPO, and AHE under a shared epoch/batch protocol on Terminal-Bench 2.0 and HLE.

Results¶

AHE (harness-level editing): +17.1pp validation, +9.1pp ID, +6.3pp OOD gain; 1.46M rollout tokens/task, 3.91M update tokens/update. Only method with positive gains across all three views.
ACE (skill/strategy memory): +2.9pp validation, +3.6pp ID, +2.5pp OOD; 1.93M rollout tokens/task. Modest but consistent transfer.
TF-GRPO (grouped rollout RL): +17.1pp validation, +3.6pp ID, −2.5pp OOD; 2.33M rollout tokens/task. Large validation gain does not transfer out-of-distribution.
Batch size (AHE sweep): batch 20 is optimal (+17.1pp validation, +9.1pp ID); batch 10 and batch 80 both regress (−14.3pp and −17.1pp validation respectively); update token cost is roughly flat (3–4M/update) across batch sizes.
Source diversity: mixed-source (Terminal-Bench + HLE) → +9.1pp ID, +6.3pp OOD at final snapshot. HLE-only → final snapshot collapses to −22.0pp ID, −21.2pp OOD despite a useful intermediate snapshot at epoch 3.
AHE replay diagnostics: final agent solves 43/80 vs. initial 34/80 train-replay tasks, but epoch 4 drops to 6/80 (rollout errors from middleware regression) before recovering; net: +13 fixed, −4 forgotten relative to \(A_0\).

Limitations¶

Evaluation instantiated on only two benchmarks (Terminal-Bench 2.0, HLE); generalization of findings to other task domains is untested.
Only three self-evolving baselines (ACE, TF-GRPO, AHE) compared; broader method coverage absent.
Process diagnostics (harness diffs, LLM-as-judge annotation) are secondary and not automatically generated — analysis depth depends on per-run manual or offline effort.
Harbor dependency constrains applicability to benchmarks that can be containerized in Harbor's task/adapter format.
OOD transfer is measured at a coarse domain level (HLE CS/AI + Engineering); finer-grained distribution shift analysis is not reported.

Relevance to Harnesses / Meta-Harnesses¶

SEAGym places the agent harness — the structured execution layer of prompts, memory, tools, middleware, and orchestration logic surrounding a base model — at the center of its formalism, making it one of the first evaluation frameworks explicitly designed to measure harness mutation over time. The paper's key insight is that harness updates are persistent, method-specific, and interaction-loop-entangled, so they require dedicated diagnostics (snapshot curves, replay, OOD transfer) beyond task-score aggregation. For researchers tracking harness and meta-harness work, SEAGym provides both a vocabulary (mutable harness state \(H_t\), update rule \(U\), evaluation views) and a reusable scaffold for comparing competing harness-evolution strategies under controlled conditions. The demonstrated failure modes — validation gain without OOD transfer, mid-run catastrophic middleware regression, source-diversity collapse — are directly relevant to anyone designing or evaluating meta-harnesses that self-modify their own execution structure.