Skip to content

EvoPolicyGym: Evaluating Autonomous Policy Evolution in Interactive Environments

🕒 Published (v1): 2026-07-02 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EvoPolicyGym introduces a controlled benchmark for Autonomous Policy Evolution, where a harness–model coding agent iteratively edits an executable decision policy under a fixed episode budget and is evaluated on held-out return of its best validation checkpoint. The benchmark isolates the iterative improvement process from open-ended software engineering noise. GPT-5.5 achieves the strongest aggregate rank score (0.891) across all 16 environments in the Core16 suite.

Problem

Existing agent evaluations either collapse iterative improvement into a single final score—obscuring failure modes like blind retries, overfitting to visible feedback, and brittle special cases—or confound it with open-ended software engineering tasks involving evolving specifications and maintenance quality. There is no controlled benchmark that isolates an agent's ability to convert bounded environment feedback into generalizable improvements of an executable decision policy.

Method

The benchmark formalizes a Autonomous Policy Evolution loop: a coding agent \(\pi_\theta\) observes workspace state \(W_i\), prior feedback \(F_i\), and remaining episode budget \(B_i\), then produces workspace patches \(u_i\) and optional submit commands \(s_i\). The formal transition is:

\[\pi_\theta(W_i, F_i, B_i, H_i) \to (u_i, s_i, H_{i+1}), \quad P_{i+1} = \Phi(W_{i+1}), \quad B_{i+1} = B_i - c_i\]

A strict visibility boundary separates train-episode feedback (visible) from validation and held-out evaluation (server-side only). The agent submits a Python Policy class implementing reset(episode_index) and act(obs), which the server runs in a sandboxed Gymnasium environment. Final scoring uses the validation-selected checkpoint's mean return on hidden held-out episodes. The Core16 suite spans 16 tasks across Gym/Box2D, MuJoCo, MiniGrid, and Robotics/Driving, each with a 128-episode training budget, 16 validation cases, and 32 held-out cases. Post-hoc best-so-far validation score trajectories are computed as diagnostics but never fed back to the agent.

Key Contributions

  • Formal definition of Autonomous Policy Evolution as a benchmarkable setting: agent iteratively revises an executable policy from bounded rollout feedback, evaluated on held-out generalization.
  • EvoPolicyGym benchmark instantiation with strict visibility boundaries, sandboxed evaluation, trajectory-level feedback, and hidden held-out scoring across Core16 (16 environments, 4 task families).
  • Trajectory-level diagnostics: post-hoc best-so-far hidden-validation score curves that reveal budget efficiency, timing of improvement events, and structural vs. parametric editing patterns.

Results

  • GPT-5.5 (Codex harness): Core16 rank score 0.891, 9 wins, top-2 on all 16 environments; leads Gym/Box2D (0.938), MuJoCo (0.875), and Robotics/Driving (0.938) families.
  • Claude Opus 4.7 (Claude Code harness): Core16 rank score 0.750, 5 wins, 12 top-2 placements; leads MiniGrid family (0.938); wins ContinuousCar, Ant, KeyCorridor, FourRooms, ObstructedMaze.
  • MiniMax-M3 (Claude Code harness): Core16 rank score 0.531; wins HalfCheetah, top-2 on Parking and FetchPickAndPlace.
  • DeepSeek-V4-Pro (Claude Code harness): Core16 rank score 0.359; wins only Roundabout, one top-2 placement total.
  • Uniform random policy baseline: Core16 rank score 0.109 (from shared zero-score ties on MiniGrid).
  • Trajectory analysis shows MiniGrid improvements are sparse but sharp; MuJoCo improvements are incremental; robotics/driving improvements are often delayed.

Limitations

  • Core16 suite covers only 16 tasks; broader adapter surface (Atari, MiniWorld, MetaWorld, BrowserGym, etc.) is not experimentally validated in this paper.
  • GPT-5.5 runs through the Codex harness while all others use the Claude Code harness—harness differences are not controlled and conflate model capability with scaffolding quality.
  • The 128-episode budget is far below conventional RL convergence regimes, so standard RL baselines are excluded; cross-paradigm comparison is unavailable.
  • Token usage and context management are not normalized across models, making compute cost comparisons unreliable.
  • Post-hoc trajectory diagnostics are reconstructed after the run; they inform analysis but are not used for mechanism attribution with causal rigor.
  • Only four models evaluated; limited statistical coverage for claims about agent behavior.

Relevance to Agentic AI / LLM Agents

EvoPolicyGym directly operationalizes a core capability question in LLM-agent research: can an agent convert bounded environmental feedback into progressively better executable policies across repeated interactions, rather than succeeding in a single shot? By enforcing a visibility boundary between train feedback and held-out evaluation, it creates a rigorous test for generalization under exploration–exploitation tradeoffs—a gap that SWE-bench and similar one-shot patch benchmarks cannot measure. The trajectory-level diagnostics (best-so-far curves, edit type analysis) provide new tools for diagnosing how agents improve, distinguishing structural synthesis from parameter tuning and early-discovery efficiency from late-recovery patterns. This is directly relevant to the growing line of work on self-improving agents (Reflexion, AlphaEvolve, Eureka) and provides a controlled substrate for benchmarking harness–model systems as integrated units.