OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents

🕒 Published (v1): 2026-05-28 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

Automated auditing of the open skill ecosystem for LLM agents; lifecycle reliability lens

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OpenSkillEval is an automatic evaluation framework that audits both skill-augmented LLM agent systems and the skills themselves across five real-world task categories. It dynamically generates task instances from evolving artifacts rather than relying on static benchmarks, and finds that skill availability does not guarantee effective skill usage—agents read provided skills in only ~48% of cases on average under realistic settings.

Problem

The open-source skill ecosystem (structured workflow instructions distilled for LLMs) has expanded rapidly, but no systematic evaluation exists to assess: (1) how different agent frameworks interact with skills during execution, (2) how to measure skill quality objectively, or (3) how to guide skill selection under cost-performance trade-offs. Static benchmarks cannot keep pace with evolving user needs, and community popularity metrics (download counts) correlate poorly with actual task performance.

Method

OpenSkillEval decomposes into three automated pipelines:

1. Artifact-driven Case Generation: For each of five task categories, high-quality existing artifacts (websites, datasets, documents) are reverse-engineered by LLMs into structured task specifications \((T)\) plus natural-language instructions \((I)\), then filtered by a verifier LLM for coherence. This produces 677 task instances generated via Claude-4.6-Opus and GPT-5.2.

2. Skill Collection: 30 community skills are curated from repositories (clawhub.ai, skills.sh, openskills.space, skillsmp.com), filtered by download count to favor higher-adoption skills.

3. Evaluation Pipeline: Two complementary views—

4. Trajectory trace analysis: Uses ATIF (Agent Trajectory Interchange Format) to normalize execution traces across heterogeneous agent frameworks (Claude Code, Codex, Gemini CLI, Kimi Code CLI); an agent-as-judge procedure decomposes each skill into intended workflow steps and scores whether agents follow, skip, or contradict each step.
5. Artifact analysis: Task-specific rubrics assess completeness, content quality, and visual design; web design uses agent-based interaction simulation; report generation and data visualization use code-based numerical accuracy checks.

The framework supports a default setting (skill placed in agent-accessible path) and a force-using setting (explicit directive injected into task prompt).

Key Contributions

• Artifact-driven dynamic benchmark generation pipeline that auto-refreshes as source content evolves, avoiding benchmark contamination.
• Unified skill collection and organization infrastructure tracking 30 open-source skills across five task domains.
• ATIF-based trajectory trace analysis enabling cross-framework comparison of skill adherence at step granularity.
• Empirical finding that skill read rate under realistic default settings averages only ~48%; Claude Opus 4.6 reads skills in only ~20% of cases.
• Evidence that force-using raises read rate to ~94% but does not eliminate autonomous step-skipping or skill contradiction.
• Practical design guidelines for skill format, length, and prior richness.

Results

• Under the default setting, agents read SKILL.md in ~48% of cases on average across frameworks; Claude Opus 4.6 reads at ~20%, Kimi K2.6 at ~75%.
• The force-using setting raises average read rate to ~94% and shifts first skill access from step 4.4 to step 3.3 in the trajectory.
• Even after reading, agents follow only a fraction of prescribed steps: step-adherence breakdowns show substantial Skip and Contra proportions across all agents (exact per-agent split visible in Figure 3b but precise overall means not stated in the provided excerpt).
• A weaker base model paired with well-designed skills and a suitable framework can match a stronger base model without skills.
• Many high-download-count community skills fail to outperform base agents without skills while incurring additional input cost.

Limitations

• Skill collection is a time-snapshot; repositories are continuously updated so findings may not generalize beyond the collection date.
• The 30-skill selection is filtered by download count, which may exclude niche but high-quality skills.
• The provided text does not include final aggregated artifact quality scores across all 677 instances, making it difficult to quantify the magnitude of skill benefit vs. baseline.
• Evaluation rubrics rely heavily on LLM-as-judge and agent-as-judge, introducing potential judge-model bias.
• The five task categories (all artifact-producing, multi-step, creative/analytical) may not generalize to instruction-following or reasoning-heavy tasks where skills are used differently.

Relevance to Harnesses / Meta-Harnesses

OpenSkillEval is itself a meta-harness: it wraps multiple heterogeneous agent frameworks (Claude Code, Codex, Gemini CLI, Kimi Code CLI) under a unified evaluation infrastructure, normalizing their execution traces via ATIF to enable apples-to-apples comparison—exactly the cross-framework orchestration abstraction that meta-harness work concerns. The artifact-driven dynamic benchmark generation pipeline is a harness-level component that automatically refreshes task instances as real-world content evolves, demonstrating how meta-harnesses can decouple evaluation stability from static dataset maintenance. The trajectory adherence analysis—decomposing skills into steps and scoring agent compliance—provides a blueprint for how meta-harnesses can introspect sub-agent behavior rather than treating agents as black boxes. For researchers building harnesses over skill-augmented agents, the finding that ~52% of agents ignore skills under default placement is a critical design constraint: harnesses cannot assume skills are consumed just because they are accessible.