Skip to content

OpenSkillEval: Automatically Auditing the Open Skill Ecosystem for LLM Agents

🕒 Published (v1): 2026-05-28 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

OpenSkillEval: systematic audit of open-source skill ecosystem for LLM agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OpenSkillEval is an automatic evaluation framework that audits both skill-augmented LLM agent systems and the skills themselves across five real-world task categories. It reveals that agents use provided skills in only ~48% of cases by default, that skill benefit depends heavily on model–framework pairing, and that many popular open-source skills fail to outperform base agents.

Problem

The open-source "skill" ecosystem (structured workflow instructions distilled for LLM agents) has grown rapidly, but there is no systematic way to assess: (1) whether different agent frameworks actually use skills during execution, (2) how skill quality varies across community-contributed skills, and (3) how to select skills given cost–performance trade-offs. Static benchmarks cannot track the evolving real-world artifact landscape.

Method

OpenSkillEval has three components:

  1. Automatic test-case generation via an artifact-driven case generation strategy: existing high-quality artifacts (websites, presentations, datasets, etc.) are reverse-engineered by LLMs into structured task specifications \((T)\) and natural-language instructions \((I)\), then validated by a verifier LLM for coherence and completeness. This yields 677 task instances across five categories: presentation generation, front-end web design, poster generation, data visualization, and report generation.

  2. Skill collection pipeline: 30 skills curated from public repositories (clawhub.ai, skills.sh, openskills.space, skillsmp.com), filtered by download count to favor community-adopted skills.

  3. Automatic evaluation pipeline combining:

  4. Trajectory trace analysis using the Agent Trajectory Interchange Format (ATIF) and an agent-as-judge procedure that decomposes each skill into intended workflow steps and checks whether agents follow, skip, or contradict them.
  5. Artifact analysis with task-specific rubrics covering completeness, content quality, and visual design; specialized checks include agent-based interaction simulation for web design and code-verified data accuracy for reports and visualizations.

Evaluated agent frameworks: Claude Code (Claude 4.6 series), Codex (GPT series), Gemini CLI (Gemini 3.1 Pro), Kimi Code CLI (Kimi K2.6), plus DeepSeek V4 Pro and GLM-5.1 integrated into Claude Code.

Key Contributions

  • A dynamic, artifact-grounded benchmark with 677 auto-generated task instances that can be continuously refreshed as source content evolves.
  • The first systematic trajectory-level analysis of skill utilization across multiple agent frameworks using ATIF and agent-as-judge decomposition.
  • An organized corpus of 30 community-contributed open-source skills enabling controlled cross-skill comparison under identical task settings.
  • Empirical findings quantifying the gap between skill availability and skill usage, the model–framework interaction effect, and the cost-performance inefficiency of many popular skills.

Results

  • Default skill read rate: agents explicitly read the provided SKILL.md in only ~48% of cases on average; Claude Opus 4.6 reads it in only ~20% of cases, despite being the strongest model tested.
  • Force-using setting: explicitly instructing agents to invoke the skill raises average read rate to 94% and shifts first skill access from step 4.4 to step 3.3 in the trajectory.
  • Step adherence (force-using): even after reading, agents skip prescribed steps and contradict instructions; proportions of follow/skip/contra vary substantially by model (exact per-model values visible in Figure 3b but not tabulated in the provided text).
  • Model–framework interaction: a weaker base model paired with well-designed skills and a suitable framework can match a stronger model; however, intrinsically weak models do not reliably benefit from skill augmentation.
  • Skill quality: many publicly popular skills (high download count) fail to outperform base agents without skills while adding input cost; Claude Opus 4.6 shows selective skill usage—read rates below 50% for data visualization and report generation but above 95% for presentation generation.

Limitations

  • The 30-skill corpus is a time-sensitive snapshot; results may not generalize to the full or future state of the ecosystem.
  • Skill quality filtering relies solely on community download counts, not ground-truth quality signals.
  • The force-using setting is an experimental control, not a realistic deployment scenario; real-world skill underutilization may be worse.
  • Task categories are limited to artifact-producing workflows; the framework does not cover conversational, reasoning-only, or tool-heavy agent tasks.
  • Exact aggregate performance numbers and statistical significance tests are not present in the provided excerpt.

Relevance to Agentic AI / LLM Agents

OpenSkillEval directly addresses a practical bottleneck in the agent capability stack: the assumption that injecting workflow knowledge (skills) into an agent's environment will improve performance. The finding that agents ignore available skills ~52% of the time by default challenges a core premise of skill-based agent augmentation and has implications for system designers relying on context-provided instructions. The agent-as-judge trajectory decomposition methodology is a reusable evaluation primitive for studying instruction-following in agentic settings more broadly, complementing work on RAG faithfulness and tool-use fidelity. The model–framework interaction effect highlights that LLM agent performance is not reducible to model capability alone, reinforcing the need for co-evaluation of backbone models and execution frameworks in agentic benchmarks.