Skip to content

Reward-Free Evolving Agents via Pairwise Validator

🕒 Published (v1): 2026-07-15 22:50 UTC · Source: Arxiv · link

Why this paper was selected

Reward-free self-evolution via pairwise validator; removes costly reward engineering from agent improvement loops

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Self-evolving agentic loops (which iteratively refine an agent's prompt or code) traditionally require a scalar reward signal that is costly to design and label. This paper replaces that reward with a frozen LLM pairwise validator that simply compares parent and child candidates and returns a binary "which is better" verdict. Across multiple agents, engines, and task substrates, the pairwise-gated variants match or exceed full-reward baselines without per-step reward design.

Problem

Scalar task rewards in self-evolving agent loops (GEPA, ADRS, ShinkaEvolve, etc.) require domain expertise and labeled examples that are expensive to assemble. In many real-world settings a reliable per-step scalar reward is hard or impossible to define. The gap is that no prior system substitutes a pairwise validator at the per-iteration accept/reject gate as a drop-in replacement for the scalar reward.

Method

At each evolution iteration the standard gate \(a_t = \mathbf{1}\{r(f_{t,c}) > r(f_{t,p})\}\) is replaced by a frozen LLM validator \(V_\varphi\) that receives the parent candidate \(f_{t,p}\) and child candidate \(f_{t,c}\) along with context \(E\) and returns a binary verdict:

\[a_t^{V_\varphi} = \mathbf{1}\{V_\varphi(f_{t,c}, f_{t,p}, E) = \texttt{better}\}\]

The gate accepts the child when it wins at least as many verdicts as the parent across a per-iteration train minibatch \(|T_t| = 3\).

Two configurations are proposed:

  • Adaptive Focus (train-reward-free): Keeps the engine's validation-set parent selection; replaces only the gate. The context \(E\) is a rubric (Accuracy / Soundness / Holistic) refreshed every 10 iterations by a meta-LLM that inspects recent gate activity, initialized at Accuracy and empirically drifting toward Holistic.
  • Soft Elo (fully reward-free): Replaces both the gate and parent selection. Candidates carry an Elo rating (initialized at 1600); each verdict triggers a confidence-weighted Elo update
\[\hat{S}_c = \frac{1}{1+10^{(R_o - R_c)/400}}, \quad R_c \leftarrow R_c + k\, c_t\bigl(S(x) - \hat{S}_c\bigr)\]

where \(c_t\) is the mean per-token probability of the agent's previous response (a proxy for output confidence), \(k=32\), and \(S(x) \in \{0, \tfrac{1}{2}, 1\}\). Parent selection uses a softmax over pool ratings. A fallback Plain Elo (\(c_t \equiv 1\), \(k=64\)) is used when per-token logprobs are unavailable.

The validator requires no training and is integrated into three existing engines (GEPA, ADRS, ShinkaEvolve) across two artifact substrates (prompt and code). The validation pool is deliberately small (\(|V| \in [12, 30]\)).

Key Contributions

  • Pairwise validator gate: A training-free frozen-LLM binary comparator replaces the per-step scalar reward at the accept/reject step of self-evolving agentic loops.
  • Adaptive Focus variant: Periodically refreshes a rubric via a meta-LLM call to steer pairwise comparison focus, dropping only train-time rewards.
  • Soft Elo variant: Fully reward-free; replaces both gate and parent selection with confidence-weighted Elo ratings derived from validator verdicts, requiring no validation-set rewards.
  • Cross-engine and cross-substrate integration: Method is validated on GEPA (prompt), ADRS (code), and ShinkaEvolve (code), covering retrieval, instruction-following, math, and competitive programming tasks.
  • Cross-family validator swap: Results transfer when the validator is swapped to Claude Haiku 4.5 or Gemma-4, ruling out self-pairing artifacts.

Results

  • Retrieval / instruction-following (GEPA, Table 2): Across 20 (model × task) cells with 5 agent models (Qwen3-8B, Qwen3-8B-thinking, Qwen3-32B, Gemma-4, gpt-oss-20b), the per-row best across pairwise variants matches or exceeds full-reward GEPA on all 20 rows. Illustrative wins: Soft Elo on HoVer (Qwen3-8B) +10.0 pp, Adaptive Focus on IFBench (Qwen3-32B) +11.6 pp, Few-shot (with rewards) on PUPA (gpt-oss-20b) +13.5 pp.
  • Math (GEPA, Table 3, robustness check): Qwen3-8B-thinking on AIME: Few-shot (with rewards), Adaptive Focus, and Plain Elo each lift accuracy from 40.0% to 60.0% (+20.0 pp, 3 problems). Other cells at parity or within one problem.
  • Validator-swap ablation (Table 4, Qwen3-8B agent): Under Claude Haiku 4.5 validator, per-row best meets or beats full-reward GEPA on all 4 retrieval/IF tasks. Under Gemma-4 validator, 3 of 4 tasks meet or beat baseline; PUPA falls short (−2.0 pp) as the lone exception.
  • Code evolution (Table 5, Qwen3-8B, ADRS + ShinkaEvolve): Pairwise gate variants match or exceed engine baselines in the majority of cells under both self-paired and cross-family (Haiku) validator setups.
  • Generalization: Pairwise variants show smaller validation-to-test gap than full-reward GEPA in the small-\(|V|\) regime (e.g., HoVer Gemma-4, AIME Qwen3-8B-thinking, IFBench Qwen3-8B).

Limitations

  • PUPA / Gemma-4 validator exception: Under the Gemma-4 cross-family validator swap, PUPA is the one task where the best pairwise variant (60.1%) falls short of the full-reward baseline (62.1%).
  • Small math test sets: AIME has 15 test problems (6.7% per problem) and LiveBench-Math has 30 (3.3% per problem), severely limiting statistical resolution on math claims.
  • Confidence weighting requires logprobs: Soft Elo's \(c_t\) term requires per-token probabilities; API-only models without logprob access must fall back to Plain Elo with a cruder update.
  • Mutator unchanged: The propose step follows each engine's default; the pairwise validator addresses only the gate and selector, not the quality of mutations themselves.
  • Self-pairing not fully eliminated: The baseline and ablations still use the same LLM for agent and validator in some cells; cross-family swaps are limited to a subset of experimental conditions.
  • Scope of engines: Integration with only three published self-evolving engines; generalization to AlphaEvolve-style or workflow-graph engines (AFlow, Darwin Gödel Machine) is not demonstrated.

Relevance to Agentic AI / LLM Agents

This work directly addresses a core bottleneck in self-improving agentic systems: the dependence on expensive scalar reward signals that prevent deployment in low-labeled-data settings. By showing that a frozen LLM pairwise judge can serve as a drop-in accept/reject gate across multiple engines and substrates, it lowers the cost barrier for applying automatic agent optimization in real-world tasks where ground-truth labels are scarce. The Soft Elo variant is fully reward-free, meaning an agent can self-evolve using only its own outputs and a judge LLM — a meaningful step toward autonomous agent refinement without human-curated benchmarks. This connects to the broader trajectory of RLAIF, LLM-as-judge, and self-play paradigms being applied not just to model training but to the outer loop of agentic scaffold optimization.