Skip to content

SkillEvolBench: Benchmarking the Evolution from Episodic Experience to Procedural Skills

🕒 Published (v1): 2026-05-22 18:23 UTC · Source: Arxiv · link

Why this paper was selected

SkillEvolBench: benchmarks distillation of episodic experience into reusable procedural skills

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SkillEvolBench is a 180-task diagnostic benchmark across six real-world agent environments that tests whether LLM agents can convert episodic task trajectories into reusable procedural skills via a host-side Skill Author call with verifier feedback. Across ten model configurations and three agent harnesses (Claude Code, Codex CLI, Gemini CLI), current agents exhibit local adaptation but not reliable reusable skill formation. Raw-trajectory reuse frequently outperforms distilled skills, revealing a lossy abstraction bottleneck in current skill-induction procedures.

Problem

Prior work conflates episodic experience reuse (replaying or retrieving past trajectories) with procedural skill formation (extracting durable, transferable procedures). SkillsBench showed curated skills help but tested self-generated skills only in cold-start, pre-execution conditions, before any trajectory or verifier feedback is available. The core open question is whether noisy one-off task experience can be distilled into compact procedural artifacts that future agents can load and apply to harder, shifted, adversarial, or compositional tasks without direct trace replay or test-time repair.

Method

Benchmark structure. Six environments covering code modification, API orchestration, data processing, document transformation, research synthesis, and communication operations. Each environment has five task families; each family defines six role-conditioned tasks: - Acquisition: canonical → enriched → variant (expose and stress-test the base procedure) - Deployment (frozen): context-shift → adversarial → composition (test transfer, shortcut resistance, multi-skill interaction)

Skill evolution protocol. After each eligible acquisition attempt, harness-recorded artifacts (tool calls, file accesses, edits, test outcomes) are compacted into a trajectory summary \(\tilde{\tau}^r_{e,f}\) and paired with structured verifier feedback \(v^r_{e,f}\). A separate host-side Skill Author call receives only same-family history \(H^{\preceq r}_{e,f}\) and the current family skill state, then emits a structured library edit: $\(L_{e,k+1} = U_c\!\left(L_{e,k},\ H^{\preceq r_k}_{e,f}\right)\)$ The update \(U_c\) can write a new skill, refine an existing one, or do nothing. The library is frozen before deployment; replay re-runs acquisition tasks post-freeze to separate local recovery from transfer.

Initialization conditions compared: - No-Skill, Raw-Trajectory (compacted traces, no abstraction) - Zero-Shot (metadata-only skill written before any execution) - Curated-Static / Curated-Revision / Curated-Always (gap-exposed human priors ± revision) - SelfGen-Experience / SelfGen-Always (induction from own trajectories)

Metrics. \(\text{LSR}\) (acquisition success rate), \(\text{RSR}\) (replay SR), \(\text{ESR}\) (frozen deployment SR), decomposed into \(\text{CSSR}\) (context-shift), \(\text{ARSR}\) (adversarial), \(\text{CompSR}\) (composition).

Key Contributions

  • SkillEvolBench benchmark: 180 tasks across 6 environments with role-conditioned arcs that enforce a strict acquisition→freeze→deployment protocol, preventing test-time skill repair.
  • Separation of four confounds: procedural abstraction, base agent capability, curated prior knowledge, and raw episodic trace reuse are disentangled via controlled conditions.
  • Host-side Skill Author architecture: a separate orchestrator call (not the task-solving agent) with access only to same-family history, enabling clean measurement of the abstraction step.
  • Empirical finding—lossy abstraction bottleneck: raw-trajectory reuse outperforms distilled skills across most configurations, showing current abstraction procedures discard contextual/procedural cues.
  • Capacity and cost analysis: more skills or larger Tier-3 resource libraries are insufficient; additional updates introduce episode-specific drift and procedural clutter.

Results

  • Skill gains are acquisition-local and unstable under frozen deployment: curated and self-generated skills sometimes improve LSR or RSR, but these gains rarely persist to frozen ESR.
  • Raw-Trajectory controls dominate distilled skills on deployment tasks across most model/harness combinations, establishing the lossy abstraction bottleneck as a consistent finding.
  • Representative LSR numbers (Claude Code / Opus 4.6): No-Skill 38.9%, Experience 44.4% (+5.5%), Always 42.2% (+3.3%); RSR: Experience 48.9% (+10.0pp over No-Skill LSR baseline).
  • Curated-Start with Always-Update (Claude Code / Opus 4.6): LSR 44.4%, CSSR 40.0%, ARSR 46.7%, CompSR 23.3%—CompSR is consistently the weakest axis across all conditions and models.
  • Gemini 2.5 Pro (weakest configuration): No-Skill LSR 30.0%; curated/self-gen variants show mixed positive and negative deltas, with CSSR dropping as low as 13.3% under some conditions.
  • More frequent authoring (Always-Update) does not reliably outperform selective-update variants; in several configurations it underperforms No-Skill on ESR.
  • Zero-Shot skills provide negligible or negative benefit over No-Skill in most configurations, consistent with SkillsBench findings.

Limitations

  • The text is truncated before the full limitations section; the following are stated or evident:
  • 180 tasks is a relatively small sample given the complexity of the claim; performance deltas of 1–3 pp are within noise for per-environment analyses.
  • Library freeze prevents adaptation: the protocol is strict by design but may underestimate hybrid approaches that allow lightweight test-time reflection without full library writes.
  • Harness-side Skill Author is a single architectural choice: the results may not generalize to integrated (in-agent) skill induction architectures.
  • Cross-environment leakage is prevented by design (library reset per environment), which may understate cumulative learning benefits in open-ended deployment.
  • Model-dependent failures: gains and losses are inconsistent across model configurations, limiting generalizable conclusions about which abstraction strategy is universally better.
  • Compacted trajectory summaries may introduce their own information loss before the Skill Author even attempts abstraction; the pipeline has two lossy stages.

Relevance to Harnesses / Meta-Harnesses

SkillEvolBench directly operationalizes a meta-harness pattern: the Skill Author is a host-side orchestrator call that sits above the task-solving agent, consuming harness-recorded artifacts and managing a persistent external library—exactly the kind of second-level control loop that meta-harness research targets. The paper's finding that raw trajectories often outperform distilled skills suggests that naive LLM-based abstraction is a weak link, and that meta-harnesses may need richer compaction strategies, structured verifier integration, or retrieval-augmented replay rather than pure distillation to achieve durable procedural transfer. The three evaluated harnesses (Claude Code, Codex CLI, Gemini CLI) serve as execution substrates whose behavioral differences propagate through the skill-evolution pipeline, making SkillEvolBench a direct testbed for how harness architecture interacts with meta-level skill management.