Skip to content

SkillFlow:Benchmarking Lifelong Skill Discovery and Evolution for Autonomous Agents

🕒 Published (v1): 2026-04-19 07:51 UTC · Source: Arxiv · link

Why this paper was selected

Benchmark for lifelong skill discovery and evolution across autonomous agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SkillFlow is a benchmark of 166 tasks across 20 workflow families that evaluates whether autonomous agents can discover, repair, and accumulate reusable skills over a lifelong learning sequence, rather than merely consume pre-provided ones. Each task family shares a Domain-Agnostic Execution Flow (DAEF) skeleton, enabling controlled measurement of cross-task skill transfer. Experiments across 11 models show that only the strongest models (notably Claude Opus 4.6, +8.43 pp) achieve reliable self-improvement, while weaker models stagnate or regress.

Problem

Existing agent benchmarks supply skills externally and test usage, not origination. No prior benchmark measures whether an agent can (1) extract reusable procedural skills from its own execution traces, (2) iteratively patch and repair those skills after failures, and (3) maintain a coherent, growing skill library across a sequential task stream of increasing difficulty within a shared workflow topology.

Method

DAEF abstraction. Each task family is built around a Domain-Agnostic Execution Flow \(F = \phi(T) = (V_F, E_F, \lambda_F)\), the abstract workflow graph obtained by stripping domain-specific grounding (\(\gamma\)) from a full task instance \(T = (V, E, \lambda, \gamma)\). Node types come from a controlled vocabulary (read, retrieve, compute, detect, output); dependency structure is preserved. This enables multiple cross-domain tasks to share an identical operational topology.

Task construction pipeline. Seed tasks from SkillsBench and GDPval are matched to \(\geq 5\) relevant skills via Qwen3-embedding-4B similarity. An Architect Agent (GPT-5.3-Codex in Cursor) generates 4 initial tasks in Harbor format conditioned on the DAEF; a Critic Agent (Claude Opus 4.6) evaluates them in real Docker environments and provides feedback. The pair iterates for up to 5 rejection rounds, then expands to 8–9 tasks per family. Human review checks instruction leakage, logical soundness, verifier correctness, and difficulty gradient.

Agentic Lifelong Learning protocol. The agent begins with an empty skill library \(S_0 = \emptyset\). For each task \(T_t\) it executes under its native harness, producing trace \(\tau_t\) and verifier-derived rubric \(r_t\). A fixed prompt template \(g\) drives skill patch generation: $\(\Delta_t = \mathrm{Model}_g(S_{t-1}, \tau_t, r_t), \quad S_t = \mathrm{Apply}(\Delta_t, S_{t-1}).\)$ Patches contain three fields: summary, upsert_files (create/overwrite SKILL.md and helper scripts), and delete_paths. Patch generation is decoupled from the task-execution harness; task execution uses each model's native agent harness (Claude Code, Codex CLI, Kimi-CLI, or Qwen-Coder). Skill use is detected by observing reads/calls to library files in execution traces.

Key Contributions

  • SkillFlow benchmark: 166 runnable tasks, 20 families, 5 broad domains (Finance & Economics, Operations & Supply Chain, Healthcare & Life Sciences, Governance & Strategy, Data & Document Intelligence).
  • DAEF framework: formal definition of domain-agnostic workflow graphs enabling controlled difficulty gradients and cross-domain task instantiation within a shared workflow topology.
  • Agentic Lifelong Learning protocol: standardized sequential evaluation where skill patches are generated purely from trajectory + rubric feedback without oracle skill injection.
  • Empirical failure-mode taxonomy: fragmented skill growth, reinforcement of erroneous logic, and the write-vs.-repair gap — cases where models produce skills but fail to reliably correct them.
  • Multi-harness evaluation: 11 model variants across 4 distinct agent harnesses, revealing harness–model interaction effects on skill evolution.

Results

  • Claude Opus 4.6: \(62.65\% \to 71.08\%\) (+8.43 pp); cost reduced (−7.52%); strongest evidence of stable self-improvement.
  • Claude Sonnet 4.5: \(49.40\% \to 55.42\%\) (+6.02 pp).
  • MiniMax M2.5: \(28.31\% \to 34.94\%\) (+6.63 pp).
  • GPT 5.4: \(33.13\% \to 36.75\%\) (+3.62 pp).
  • Kimi K2.5: \(55.42\% \to 56.02\%\) (+0.60 pp) despite 66.87% skill usage — high usage does not imply high utility.
  • GPT 5.3 Codex: \(52.41\% \to 46.39\%\) (−6.02 pp); skill evolution hurts.
  • Qwen-Coder-Next: \(45.18\% \to 44.58\%\) (−0.60 pp); regresses despite 12.05% skill usage.
  • Claude Opus 4.6 history-prepend control: 51.04% (below both vanilla 62.65% and skill-evolution 71.08%), confirming that skill library structure matters beyond raw context injection.
  • Domain breakdown: Finance & Economics shows more negative transfer; Data & Document Intelligence more often shows positive transfer.

Limitations

  • Lifelong evaluation is within-family only (family reset between domains); cross-family skill transfer is explicitly excluded to avoid retrieval confounds, limiting ecological validity for open-ended deployments.
  • 11 models evaluated but harness–model pairings are not fully crossed (each model is paired with its "matched" harness), making it hard to isolate harness vs. model contributions.
  • Skill patch generation relies on model's native capability with a fixed prompt template; prompt sensitivity is not ablated.
  • The benchmark is English-only and restricted to command-line/document-processing tasks; generalization to GUI or embodied agents is untested.
  • Family sizes of 8–9 tasks may be too short for skill libraries to mature, potentially underestimating lifelong learning potential.
  • No forgetting metric: the protocol does not formally measure catastrophic forgetting or skill library bloat over longer sequences.

Relevance to Harnesses / Meta-Harnesses

SkillFlow is directly structured around the interplay between agent harnesses and skill libraries: each of Claude Code, Codex CLI, Kimi-CLI, and Qwen-Coder serves as an execution harness, and the benchmark explicitly decouples task-execution harness logic from skill-patch generation logic — a clean meta-harness design pattern. The DAEF itself is a meta-level workflow abstraction (a harness schema) that governs how tasks are templated and sequenced, making SkillFlow a testbed for how well harness-level workflow structure transfers across instantiated tasks. The finding that skill evolution outcomes vary substantially across harness–model pairs (e.g., Claude Code + Opus 4.6 gains +8.43 pp while Codex CLI + GPT 5.3 Codex loses −6.02 pp) motivates further research into how harness design choices — skill-read signaling, patch interfaces, context management — modulate lifelong learning. For harness builders, the upsert_files / delete_paths patch schema and the separation of skill generation from task execution offer a reusable architectural template.