Skip to content

SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use

🕒 Published (v1): 2026-07-02 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SkillCoach introduces a self-evolving rubric framework that decomposes agentic skill-use into four trajectory-level dimensions—selection, following, composition, and reflection—kept separate from binary verifier success. Rubrics are bootstrapped from gold skills and real rollouts, then iteratively refined through validation-gated arbitration patches. The evolved rubrics both expose failures hidden by final accuracy and serve as a stronger process-supervision filter for SFT than outcome-only trajectory filtering.

Problem

Final verifier success is too coarse a signal for evaluating or training skill-using LLM agents. In realistic enterprise skill repositories, agents may pass the verifier through trial-and-error—selecting distractor skills, skipping required steps, composing workflows in the wrong order, or omitting final validation checks—yet these behaviors are indistinguishable from reliable, reusable skill-use under outcome-only evaluation. No prior framework provides process-level rubrics that are grounded in the structure of the skill library itself.

Method

Task setup. Skill-dependent tasks are filtered via three criteria on five agent backends: no-skill success \(p_{\text{no}} \leq 0.30\), gold-skill gain \(p_{\text{gold}} - p_{\text{no}} \geq 0.40\), and key-step coverage \(c_{\text{key}} \geq 0.70\). Each task is paired with a distractor-augmented library \(L_t = G_t \cup D_t\) (2 unrelated + 3 semantically similar inapplicable skills).

Four-dimensional meta-ability. Each trajectory \(z_t\) is scored along: - Skill selection \(s_{\text{sel}}\): set-level F1 over gold vs. distractor skills, requiring visible evidence of skill reads/invocations. - Skill following \(s_{\text{fol}} = \sum_k w_k c_k(z_t) m_k(z_t) / \sum_k w_k\), where \(m_k \in \{0,1\}\) is an evidence multiplier preventing credit for unsupported claims. - Skill composition \(s_{\text{comp}}\): fraction of precedence dependencies \((u,v) \in P_t\) correctly satisfied with artifact transfer. - Skill-grounded reflection \(s_{\text{ref}}\): weighted coverage of expected output checks before submission.

Self-evolving rubrics. An initial rubric \(R^0\) is derived from gold skills, distractor boundaries, and representative rollouts. At each evolution round, an arbitration LLM proposes a local patch \(\tilde{R}^{j+1} = R^j \oplus A_\psi(R^j, C_t)\) (using calibration rollouts \(C_t\)). The patch is accepted on held-out validation rollouts \(U_t\) only if it passes a hard gate \(\Delta H \geq 0\) (no destructive changes: cannot ignore distractors or mark unsupported steps as correct) and a soft objective \(\Delta Q > \epsilon\) with material improvement. The best accepted rubric \(R^{\text{best}} = \arg\max_{R \in A_t} Q(R, U_t)\) is selected.

Rubric-filtered SFT. Training trajectories are retained only when rubric meta-score \(\geq 0.95\) and the external verifier passes, producing dataset \(D_{\text{sft}}\).

Key Contributions

  • First formulation of agentic skill-use as a four-dimensional trajectory-level meta-ability (selection, following, composition, reflection), explicitly decoupled from the external verifier.
  • SkillCoach: first self-evolving rubric framework with evidence-grounded judging, local arbitration patches, and validation-gated acceptance under distractor-augmented skill libraries.
  • Dual-use of evolved rubrics: diagnostic evaluation instrument and process-supervision filter for SFT.
  • Skill-dependency filtering protocol that selects tasks where gold-skill availability causally matters (gain \(\geq 40\) pp over no-skill baseline).

Results

Rubric quality (Table 2, \(R^0\) vs. \(R^{\text{best}}\), 50 paired instances, 10 held-out task families, Gemini 3.1 Pro as independent judge): - Gold-keypoint coverage: 71.56 → 83.70 (+12.14 pp) - Usability score: 81.53 → 94.33 (+12.80 points) - Hallucination rate: 2.00 → 0.00 (−2.00 pp) - Trajectory-filtering consistency: 82.00 → 96.00 (+14.00 pp)

Skill-use evaluation (Table 3, gold + distractor setting): - Opus 4.7 final accuracy: No Skills 18.0 → Gold Skills 88.0 (+70 pp), confirming skill-dependency. - GPT-5.5: 24.0 → 80.0 (+56 pp). - Gemini 3.1 Pro skill-selection score drops from 98.0 (gold only) to 78.0 (gold + distractors), despite maintained following score—isolating selection as the failure mode. - Qwen3.5-9B selection drops from 92.0 to 44.0 with distractors, accompanied by lower reflection and final accuracy.

(Full training/SFT comparison results are in the truncated portion of the paper and not available here.)

Limitations

  • Small task scale: 18 training + 10 test task-family splits; generalization to diverse enterprise domains is untested.
  • Rubric quality validation uses an LLM judge (Gemini 3.1 Pro), which is itself imperfect and may share biases with rubric generation.
  • Distractor configuration is synthetic and fixed (2 unrelated + 3 semantically similar); real enterprise repositories may have different overlap distributions.
  • The validation-gated evolution depends on sufficient calibration and validation rollout quality, which degrades for rare or high-difficulty tasks.
  • SFT filtering threshold (meta-score \(\geq 0.95\)) is a fixed heuristic not tuned across agent types.

Relevance to Agentic AI / LLM Agents

SkillCoach directly addresses a critical gap in the agentic AI stack: the assumption that skill retrieval guarantees skill-use correctness. By formalizing skill-use as a four-dimensional process meta-ability with concrete scoring functions, it provides a principled alternative to outcome-only RL/SFT that is directly applicable to enterprise agent deployments using SOP-encoded skill libraries. The self-evolving rubric mechanism is closely related to the broader line of work on process reward models (PRMs) and step-level supervision for agents, offering a scalable way to generate task-specific process criteria without per-task human annotation. For researchers tracking LLM agents, this paper is notable for explicitly separating process quality from verifier success—a distinction that matters for diagnosing agent failures, designing training curricula, and deploying agents reliably in production skill repositories.