Skip to content

SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents

🕒 Published (v1): 2026-07-17 01:55 UTC · Source: Arxiv · link

Why this paper was selected

First corpus + eval of SKILL.md ecosystem; ground truth on skill quality and reuse for real-world agent harnesses

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SkillCorpus aggregates ~821,000 community SKILL.md files, curates them through a six-stage pipeline into 96,401 quality-gated, licence-cleared skills, and pairs them with a fine-tuned retrieval-and-selection stack. Evaluated across three real-world agent benchmarks, two harnesses, and open-to-frontier backbones, integrating SkillCorpus yields consistent positive gains (largest +7.5 pp on SkillsBench), with gains bounded by corpus coverage and harness compatibility.

Problem

The open SKILL.md ecosystem—procedural knowledge files that extend LLM agent capabilities at inference time—is fragmented across thousands of repositories, heavily duplicated, and uneven in quality. No prior work had produced (1) a single broadly aggregated, quality-gated, OSI-permissive corpus at ecosystem scale, or (2) a rigorous end-to-end evaluation of community skills on real-world agent tasks under a deployable retrieval stack (prior evaluations used oracle hand-picked skills, simulated environments, or single harnesses over limited sources). Uncurated community libraries had also been shown to fail to improve over a no-skill baseline.

Method

Aggregation and curation. A registry of 62 sources (GitHub repos, awesome-lists, marketplace APIs, sitemaps, JSON catalogs) yields ~821,000 raw SKILL.md files from 25,159 repositories, passed through a six-stage funnel: 1. Parse (structural conformance) 2. Length/form filter 3. Two-tier deduplication: exact content/name fingerprint collapse (−59.7%), then semantic embedding (1024-dim) at cosine ≥ 0.90 with LLM adjudication for borderline pairs (0.90–0.995), collapsing 283,844 → 101,111 skills 4. LLM judge assigning a three-facet quality score across utility \(u\), robustness \(r\), safety \(s \in \{0,\ldots,10\}\) with a 19-flag taxonomy (5 hard-gate flags triggering zero score) 5. Safety hard-gate + OSI-permissive licence filter 6. Source-prior shrinkage, retrieval embedding, and index attachment

The composite quality score is: $\(\text{content\_q} = 0.50u + 0.35r + 0.15s\)$ attenuated by \(0.5 + 0.5(s-0.3)/0.4\) when \(s \in [0.3, 0.7]\), then: $\(\text{score} = 0.85\,\text{content\_q} + 0.15\,\text{prior}_\text{src} + b\)$ where \(b\) is a small structural bonus (+0.05 for scripts/, +0.02 for references/), clipped to \([0,1]\).

Retrieval-and-selection stack. Candidate recall uses Qwen3-Emb-0.6B fine-tuned on the deduplicated corpus; Qwen3-Rank-0.6B reranks top candidates. An LLM selector gate reads full skill bodies of reranked candidates and injects 0–2 skills per task. An optional LLM query rewriter normalises jargon to corpus vocabulary.

Evaluation. Four (harness × backbone) cells: OpenClaw and Raven harnesses × Qwen3.5-27B and Qwen3.5-397B-A17B-GPTQ-Int4, plus a single-cell frontier check on Claude Opus 4.7. Three benchmarks: SkillsBench (87 tasks, pass@1), GDPVal (220 tasks, GPT-4o-judged reward), QwenClawBench (100 tasks, hybrid score). Each cell evaluated under no-skill vs. SkillCorpus conditions averaged over three runs (74 total end-to-end runs).

Key Contributions

  • End-to-end framework unifying multi-source aggregation, six-stage curation, fine-tuned retrieval, and real-world evaluation of the open SKILL.md ecosystem
  • 96,401-skill corpus with three-facet quality scores, 16-class taxonomy, and precomputed 1024-dim embeddings, fully OSI-permissive and redistribution-cleared
  • Fine-tuned retrieval-and-selection stack (Qwen3-Emb-0.6B + Qwen3-Rank-0.6B + LLM selector) released as engineering tooling alongside the corpus
  • Empirical characterization of two operational boundaries—coverage boundary (task outside corpus scope) and harness boundary (harness skill-injection architecture)—that determine when community skills help or do not

Results

  • SkillsBench (87 tasks): pooled \(\Delta = +7.5 \pm 2.3\) pp (\(z=3.2\)); per-cell range +4.2 to +13.4 pp; strongest cell (Raven × Q-397B): 9.2% → 22.6%; frontier check (Claude Opus 4.7): 39.1% → 47.1% (+8.0 pp)
  • GDPVal (220 tasks): pooled \(\Delta = +1.51 \pm 0.49\) pp (\(z=3.1\))
  • QwenClawBench (100 tasks): pooled \(\Delta = +2.79 \pm 0.70\) pp (\(z=4.0\))
  • Industrial & Physical category on SkillsBench: +45.5 pp (lowest-baseline subcategory)
  • No cell shows net-negative mean performance; corpus behaves as a no-harm attachment at cell-mean level
  • Standalone retrieval vs. SkillRouter: matches on Easy tier, exceeds on Hard tier (within 1.7 pp Recall@10), +10.7 pp Hit@1 on SkillCorpus pool
  • Ablation (Raven × Q-397B, SkillsBench): full pipeline 22.6%, off-the-shelf retrieval 13.8%, raw crawl 14.9%, no-skill baseline 9.2%—both curation and retrieval contribute independently

Limitations

  • Ablation results are single-run only; inter-arm gaps of a few tasks are treated as directional, not statistically robust
  • Frontier evaluation covers only one cell (OpenClaw × Claude Opus 4.7 on SkillsBench); a full frontier grid is deferred to follow-up work
  • High-baseline benchmarks (GDPVal, QwenClawBench at 70–85%) exhibit ceiling effects and high per-task judge noise (reward std 20–27 pp), making individual-cell gains indistinguishable from zero
  • Facet construct validity is not confirmed by independent human gold annotation; correlation structure is read qualitatively
  • Corpus is software- and data-leaning (Dev + Data + Writing + DevOps-Infra > 50%), potentially underrepresenting other task domains
  • "Deployable" is a packaging claim; no production-environment validation is reported
  • Harness boundary effect is identified but not fully decomposed (injection mechanism vs. skill-context utilization)

Relevance to Agentic AI / LLM Agents

SkillCorpus directly operationalizes the skill layer of the emerging layered agent architecture—where a frozen backbone is augmented with externalized, retrievable procedural knowledge—and provides the first ecosystem-scale empirical account of when this augmentation actually improves real-world task outcomes. The finding that even Claude Opus 4.7 gains +8.0 pp from community skills on SkillsBench refutes the assumption that frontier models make external skill pipelines redundant. The identification of the harness boundary as a key modulator of gain has direct design implications for agent scaffold developers choosing how to integrate skill injection. Methodologically, the two-tier deduplication and three-facet quality framework offer a transferable blueprint for curating any community-contributed agent knowledge base (tools, memory snippets, MCP server catalogs) at scale.