Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents¶

🕒 Published (v1): 2026-07-14 14:02 UTC · Source: Arxiv · link

Why this paper was selected

Self-improving agents that co-evolve their own evaluation metrics; solves the hidden-metric-assumption problem

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Self-improving LLM agent loops assume a reliable evaluation metric exists, but in most real deployments one does not. This paper proposes co-evolving the evaluation metric alongside the skill library — a system called Double Ratchet — using anchored search over compositions of atomic drawback detectors, recovering 88–110% of oracle-driven lift without ground truth. It further demonstrates a complete Goodhart failure-detection-and-repair cycle on a reference-free task.

Problem¶

Self-evolving agent systems (e.g., Voyager, Ratchet) require an evaluation signal to decide pass/fail for each task attempt. On code benchmarks, unit tests supply this for free; in deployed enterprise settings no accurate automatic verifier exists, and the industry workaround is manual rubric writing or human demonstrations. Existing metric-learning approaches (reward modeling, LLM-as-judge) either require large labeled sets or share failure modes with the solver they grade, enabling Goodhart collapse under co-optimization.

Method¶

Metric representation. A metric is an expression tree over a pool of drawback detectors (atomic ops) following the grammar: $$e ::= o \;\Big|\; \bigvee_{i=1}^{k} e_i \;\Big|\; \bigwedge_{i=1}^{k} e_i \;\Big|\; \sum_{i=1}^{k} v(e_i) \geq K$$ Ops are typed (static parse/inspect, execution-based, or narrow LLM judge), each checking exactly one failure class; verdicts are cached, making expression evaluation a table lookup.

Metric loop. Each round: (1) sense — identify misses on the 10-item anchored dev set and abstention gaps on unlabeled train outputs; (2) grow — cluster failures by class, synthesize a typed op spec, admit via a birth gate (fires on ≥50% of its cluster, clean on known-good); new ops from unlabeled gaps enter as shadows (non-selectable) until they demonstrably raise dev agreement; (3) select — choose the expression maximizing $S(e) = A_\text{dev}(e) \cdot A_\text{train}(e)^w - \lambda C(e)$ subject to a validity gate; (4) curate — retire ops with non-positive leave-one-out marginal contribution.

Double Ratchet co-evolution. The metric loop and the Ratchet skill loop alternate under a front-loaded curriculum (metric phases longest early). The evolved metric grades the skill loop's training tasks; the skill loop's held-out evaluation is pinned to the locked test set the metric never reads, so a corrupted metric cannot corrupt measurement.

Anchor discipline (anti-Goodhart). Three splits: large unlabeled train, 10-item anchored dev (sole supervised signal), and locked held-out test (never read by any loop). Anchor guards include fail-closed selection (a candidate with no dev opinion is unselectable) and a validity gate dropping always-pass/fail/abstain metrics.

Independent final judge. For reference-free tasks, a stronger LLM judge (Claude Opus 4.8) compares final outputs pairwise against the pre-evolution baseline with position swapping, serving as an outer audit outside all loops.

Key Contributions¶

Formulates evaluation metric evolution as anchored search over compositions of atomic drawback detectors under a full lifecycle (birth gate, shadow promotion, merit-based retirement).
Introduces a leakage-safe three-split protocol with a selection objective combining tiny-anchor agreement ($A_\text{dev}$) and unlabeled consensus regularization ($A_\text{train}$); ablations isolate that anchor guards (not the lifecycle) prevent metric collapse.
Double Ratchet: co-evolution of metric and skill loops recovering 88–110% of oracle-driven lift on MBPP+, Spider 2.0-Snow, and reference-free report generation.
Documents an end-to-end Goodhart detection, repair, and judge-auditing episode: evolved skills gamed a rubric tag counter, one detector repaired it, and a task-aware re-judging moved the win rate from 0.515 to 0.770.

Results¶

MBPP+ (code generation): Double Ratchet peak 0.717 ± 0.038 vs. oracle 0.700 ± 0.025; lift retention 106%; $\Delta$peak CI $[-0.03, 0.06]$ (contains zero).
Spider 2.0-Snow (text-to-SQL): peak 0.458 ± 0.038 vs. oracle 0.483 ± 0.038; lift retention 110%; $\Delta$peak CI $[-0.08, 0.03]$ (contains zero). Evolved metric agrees with ground truth at only $0.500 \pm 0.026$ yet retains full oracle lift, demonstrating directional grading suffices.
Report generation (reference-free): lift retention 88% against best-available RAQS rubric; $\Delta$peak CI $[-0.05, -0.03]$ (small gap).
Goodhart repair 2×2: pre-repair win rate under task-aware judge = 0.515 (gamed proxy); post-repair = 0.770. Generic judge masked the fix (0.122 → 0.126).
Anchor-guards-off ablation: metric collapses to a vacuous detector; co-loop's apparent higher score reflects unfiltered practice, not functional grading.
No-lifecycle ablation: anchor guards without lifecycle still yields a valid (non-vacuous) metric; lifecycle contributes efficiency, not safety.
10-item dev set is not fragile: offline re-sampling over 200 stratified subsamples at sizes 4–10 gives held-out agreement $0.854$–$0.882 \pm 0.041$–$0.063$.

Limitations¶

Three seeds and 40–48-item held-out sets support only coarse statistical conclusions; CIs are wide.
Solver is a single LLM call with no tools or chain-of-thought; absolute scores are not comparable to leaderboard agents.
Teacher quality (which labels the 10-item anchor) is held fixed; its influence on anchor quality is unexamined.
Spider 2.0-Snow metric achieves only ~0.50 anchor agreement, bounding the method's added value to tasks with mechanically detectable failures; the approach provides less benefit where failures require live exploration.
The method requires a human to supply ten anchor examples with golden references — still a manual step, albeit minimal.
Report generation rubric (RAQS) is a partial reference, not ground truth; the final judge (itself requiring auditing) is the decisive signal there.

Relevance to Agentic AI / LLM Agents¶

This paper directly addresses the bottleneck that limits real-world deployment of self-improving LLM agent loops: the absence of reliable automatic verifiers. By treating the evaluator as a first-class evolving object with the same lifecycle discipline applied to skill libraries, it extends the self-improvement paradigm to settings where ground truth is unavailable — which is the common case outside code benchmarks. The Goodhart episode and repair protocol are practically important for any agentic system that co-optimizes against an imperfect metric, demonstrating that failure-expecting architecture (independent outer audits, anchor guards, inspectable metrics) is necessary rather than optional. The directional-grading insight — that a metric with only 50% accuracy can still guide a skill loop to oracle-level lift — is a useful bound for practitioners designing evaluation infrastructure for deployed agents.