Skip to content

EvoTest: Evolutionary Test-Time Learning for Self-Improving Agentic Systems

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Evolutionary test-time learning enables agents to acquire complex skills on the fly

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EvoTest is a gradient-free, test-time learning framework that improves LLM agents across repeated episodes of the same task by evolving their entire configuration — prompt, memory, hyperparameters, and tool-use routines — rather than updating model weights. It is evaluated on J-TTL, a new benchmark built on Jericho text-adventure games, where the agent must improve its score over \(K\) consecutive playthroughs of the same game. EvoTest achieves 38% higher AUC than the best prompt-evolution baseline and 57% over online RL, and is the only method to win any games.

Problem

Current LLM agents deploy with fixed configurations and cannot adapt on the fly during a test session. Existing adaptation methods — reflection, memory augmentation, and online RL/SFT — each fail in characteristic ways: reflection does not alter decision-making logic; memory does not teach new behaviors; online fine-tuning is too data-inefficient for sparse-reward, short-session settings. No standardized benchmark existed to measure rapid, in-session agent improvement.

Method

EvoTest frames agent adaptation as whole-system evolution over a sequence of episodes. The learnable configuration is \(\chi = (p, M, h, u)\): a policy prompt \(p\), a structured deployment-time memory \(M\) (success/failure tables of state–action pairs), inference hyperparameters \(h\) (e.g., temperature, exploration strength), and tool-use routines \(u\) (memory query logic and an evolvable Python state-abstraction function).

Two agents operate in an Act–Evolve loop: 1. Actor Agent plays episode \(e\) under the current configuration \(\chi^{(e)}\), producing trajectory \(\tau^{(e)}\). 2. Evolver Agent (powered by o3) reads \(\tau^{(e)}\) and produces a set of child configurations \(C^{(e+1)}\) via four operators: prompt mutation, memory update (parsing score-delta events), hyperparameter tuning, and tool-use refinement.

Configuration selection for the next episode uses Upper Confidence Bound (UCB) over the parent and all children: $\(\chi^{(e+1)} = \arg\max_{\tilde{\chi} \in \{\chi^{(e)}\} \cup C^{(e+1)}} \left[ \hat{\mu}(\tilde{\chi}) + \beta \sqrt{\frac{\log N}{1 + n(\tilde{\chi})}} \right]\)$ This balances exploitation of proven configurations with exploration of new mutations, and lets the system revert to the parent if a child underperforms.

Key Contributions

  • J-TTL benchmark: A test-time learning evaluation using six Jericho Interactive Fiction games; performance measured by per-episode AUC normalized to \(R_{\max}\) over \(K\) episodes.
  • EvoTest framework: Gradient-free, whole-system agent evolution (prompt + memory + hyperparameters + tool-use routines) driven by transcript-level LLM analysis.
  • UCB-based configuration selection: Principled exploration–exploitation balancing across an evolving population of agent configurations.
  • State-of-the-art results: +38% AUC over the strongest prompt-evolution baseline (EvoPrompt); +57% over online GRPO; only method to win any games (Detective and Library).

Results

  • Average AUC (Gemini-2.5-flash backbone): EvoTest 0.47 vs. EvoPrompt 0.34, Reflexion 0.32, GRPO 0.26, Static 0.11.
  • Average AUC (Claude-4-sonnet backbone): EvoTest 0.50 vs. EvoPrompt 0.36, Reflexion 0.34, GRPO 0.30, Static 0.12.
  • Detective (Rmax=360): EvoTest AUC 0.94/0.95 vs. Reflexion 0.58/0.60, EvoPrompt 0.65/0.67.
  • Library (Rmax=30): EvoTest AUC 0.77/0.80 vs. EvoPrompt 0.48/0.50, Reflexion 0.41/0.44.
  • EvoTest is the only method to win Detective and Library; all baselines fail to win any game.
  • Learning curves show steeper and more stable improvement trajectories than all baselines across all six games.

Limitations

  • Evaluated exclusively on text-based Interactive Fiction games (Jericho); generalizability to other agentic domains (web, code, robotics) is untested.
  • Requires a powerful separate Evolver LLM (o3) for the evolution step, adding inference cost beyond the Actor's backbone.
  • UCB exploration assumes repeated evaluation of the same task; the method is not designed for single-episode or distribution-shift settings.
  • Weight-update baselines (SFT, GRPO) use a different backbone (Qwen3-32B) rather than the same API model, complicating direct comparison of gradient-based vs. gradient-free approaches.
  • Performance on very sparse games (Zork1, AUC 0.14–0.16) remains low even with EvoTest, suggesting limits when rewards are extremely rare.

Relevance to Agentic AI / LLM Agents

EvoTest directly addresses one of the most fundamental open problems in agentic AI: enabling agents to self-improve at test time without weight updates, which is critical for deployment in novel or dynamic environments. By treating the entire agent configuration — not just the prompt — as the unit of evolution, it goes beyond Reflexion-style reflection and prompt-optimization baselines to achieve holistic, multi-faceted adaptation. The UCB-based configuration selection introduces principled bandit-style reasoning into the agent's self-improvement loop, a technique previously rare in agentic self-evolution literature. The J-TTL benchmark also fills a notable gap by providing a rigorous, reproducible setup for measuring in-session learning, which will be directly useful to researchers developing adaptive LLM agents.