DemoEvolve: Overcoming Sparse Feedback in Agentic Harness Evolution with Demonstrations¶

🕒 Published (v1): 2026-05-23 12:10 UTC · Source: Arxiv · link

Why this paper was selected

Harness evolution via demonstrations; sparse-feedback sample-efficient adaptation paradigm

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

DemoEvolve addresses the failure of self-rollout harness evolution in long-horizon stochastic environments by augmenting the coding proposer's information with competent human demonstration trajectories. Instead of retraining model weights, it evolves the external executable harness (prompts, state tracking, tools, action filtering, control flow) around a frozen LLM using human play as diagnostic reference. On Balatro, a poker roguelite, DemoEvolve achieves 12/15 rollout completions versus 6/15 for the best self-rollout baseline.

Problem¶

Prior harness evolution methods (e.g., Meta-Harness) rely on self-generated rollouts to guide a coding proposer in rewriting the agent's external harness. This works in short-horizon or model-native tasks where failures are local and attributable, but breaks in long-horizon stochastic environments: terminal rewards are sparse and high-variance, early decisions compound across hundreds of state transitions, and same-seed rollout variance causes the proposer to select inactive or noncausal harness edits that appear beneficial by chance—a noisy-selection failure mode.

Method¶

DemoEvolve defines harness evolution as optimization over executable programs $h \in \mathcal{H}_T$ around a frozen model $\pi_\phi$: $$h^* = \arg\max_{h \in \mathcal{H}_T} \mathbb{E}_{x \sim \mathcal{X}_T,\, \tau \sim p_{\mathcal{W}_T, \pi_\phi, h}(\cdot|x)} [R_T(\tau, x)]$$

A coding proposer $P$ (separate from the task-execution agent) iterates over a filesystem archive $D_t$ of prior harness code, rollout trajectories, execution traces, and scores. At each outer-loop iteration $t$, $P$ proposes $k_t$ candidate harnesses; each is evaluated and its directory appended to $D_t$.

DemoEvolve adds human demonstration trajectories $\mathcal{T}^\text{human} = \{(\tau_j^H, R_T(\tau_j^H, x_j))\}_{j=1}^m$ to the proposer's context, where $\tau_j^H = (o_0^H, a_0^H, o_1^H, \ldots)$ uses the same observation-action format as agent rollouts. Demonstrations serve as behavioral exemplars and diagnostic references—not as supervised labels or fine-tuning data—allowing the proposer to compare failed agent rollouts against competent states and localize missing abstractions, action priorities, or resource-management logic.

Three information regimes are compared: - Meta-Harness ($I_t^\text{meta} = D_t$): self-rollout archive only - OpenResearch ($I_t^\text{open} = D_t \cup \mathcal{K}^\text{text}$): archive + external textual knowledge (rules, tutorials, strategy guides) - DemoEvolve ($I_t^\text{demo} = D_t \cup \mathcal{T}^\text{human}$): archive + human trajectories

Key Contributions¶

DemoEvolve: a demonstration-bootstrapped harness evolution method that uses human trajectories as proposer-side reference experience for credit assignment in sparse-feedback regimes.
Noisy-selection failure mode: formal characterization and empirical evidence that self-rollout evolution can select inactive or noncausal harness edits due to same-seed rollout variance and trajectory drift in long-horizon tasks.
Auditing methodology: rollout replay with additional logging to verify whether selected harness edits are functionally active (hooks trigger, state variables are computed, model-call inputs change)—distinguishing genuine functional improvement from lucky selection.
Strategy transfer evidence: attrition-adjusted shop-entry economy curves showing that DemoEvolve's evolved harness produces human-like resource-management behavior, not merely score noise.
Liar's Dice positive control: macro held-out reward improves from 0.392 (base) to 0.800 (evolved) across six model/variant pairings, confirming self-rollout evolution works when failures are local.

Results¶

Liar's Dice (short-horizon positive control): - Macro held-out reward: base 0.392 → evolved 0.800 (+0.408 absolute) - Small3: 0.400 → 0.828; OneCall-Wild1: 0.383 → 0.772 - All 6 clean run-level comparisons improve on held-out seeds

Balatro (long-horizon stochastic): - BalatroLLM baseline: 5/15 completions, overall mean round 16.93 - Meta-Harness: 6/15 completions (16.80–18.13 mean round); apparent ID gain (+2.33 mean round) does not transfer OOD (−0.50); audit reveals selected candidate has a never-triggered implementation bug - OpenResearch: 4/15 completions, 16.80 overall mean round—worse than Meta-Harness overall - DemoEvolve: 12/15 completions, 22.00 overall mean round; 8/9 ID, 4/6 OOD

Economy/behavioral analysis (Balatro): - Human-trajectory distance (attrition-adjusted economy curve): Meta-Harness 30.83, OpenResearch 24.03, DemoEvolve 12.26 (late-game: 6.49 vs. 54.89 for Meta-Harness) - DemoEvolve late reach: 0.81 vs. 0.32–0.45 for other evolved conditions

Limitations¶

Experiments cover only two games (Liar's Dice, Balatro); generalization to open-ended real-world tasks or robot environments is unverified.
Human demonstration collection is not automated—the method requires competent human play, which may be costly or unavailable for novel task worlds.
The localization analysis (5×3 diagnosis/design study) is presented as a separate controlled study; its relationship to full evolution run outcomes is not fully quantified in the excerpt.
Fixed-seed evaluation with small rollout counts (3 per seed, 5 seeds) limits statistical power; completion-rate granularity is coarse.
DemoEvolve is compared against text-knowledge (OpenResearch) rather than other human-feedback approaches (e.g., RLHF, preference labels), so the specific advantage of trajectory demonstrations over other forms of human knowledge is incompletely isolated.

Relevance to Harnesses / Meta-Harnesses¶

DemoEvolve directly extends the Meta-Harness paradigm—treating the executable agent harness (prompts, tools, state tracking, control flow) as the optimized object around a frozen LLM—and identifies a concrete failure mode of that paradigm in long-horizon stochastic settings: the proposer's inability to assign credit from sparse terminal rewards. The paper formalizes the outer-loop harness search loop, distinguishes score improvement from functional harness improvement via rollout auditing, and proposes demonstrations as a practically available signal to stabilize the proposer's diagnostic credit assignment. For researchers tracking harness/meta-harness work, this paper establishes that human trajectory grounding is necessary (not just text knowledge) when self-rollout signals are too noisy, and introduces behavioral and audit methodology that could generalize to harness evolution in any stochastic long-horizon domain.