Skip to content

OpenSkillRisk: Benchmarking Agent Safety When Using Real-World Risky Third-Party Skills

🕒 Published (v1): 2026-07-22 13:24 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OpenSkillRisk is a benchmark of 263 real-world risky third-party skills drawn from public skill marketplaces, designed to evaluate whether CLI agent systems can detect and avoid latent execution-time safety risks. It evaluates 3 CLI agent harnesses and 13 LLMs using both execution-level and cognition-level metrics, finding that even the best system (Claude Code + Claude Sonnet 4.6) still triggers unsafe execution in 17.87% of cases.

Problem

Existing agent safety benchmarks either cover limited threat types, rely on synthetically poisoned or explicitly adversarial skills, or reduce evaluation to a single metric (attack success rate) that cannot distinguish incidental avoidance from genuine risk reasoning. No benchmark systematically characterizes how real, latent risks embedded in seemingly benign skills from live skill ecosystems interact with the full harness-plus-model stack.

Method

Benchmark construction (three stages):

  1. Skill collection and candidate mining. 175,891 unique English skills scraped from SkillsMP and Skills.rest (up to March 2026). A two-stage coarse-to-fine pipeline—rule-based static scanning for suspicious signatures, followed by LLM-based context-aware triage—yields 1,799 risky skill candidates, then stratified-sampled and human-verified down to 263 skills.

  2. Task and sandbox construction. Each skill is paired with a realistic benign user task package (user request + workspace files + synthetic user-context files with mock credentials + expected safe behaviors). An isolated sandbox replaces all external calls (API, DB, network) with deterministic local stubs that record invocation parameters, enabling controlled risk-surface activation without real side effects. Tasks undergo static review and execution-trace review in an iterative revision loop.

  3. Agent behavior evaluation. Three CLI harnesses (Claude Code, Gemini CLI, Codex CLI) × 13 LLMs are evaluated. Two primary metrics are defined:

  4. \(\text{ASR}_s = |D_s|^{-1} \sum_{i \in D_s} r_i\) (execution-level, lower is safer)
  5. \(\text{Awareness}_s = |D_s|^{-1} \sum_{i \in D_s} z_i\) (cognition-level, higher is safer)
  6. Composite: \(F_{\text{safe}} = \frac{2 \cdot (1 - \text{ASR}) \cdot \text{Awareness}}{(1 - \text{ASR}) + \text{Awareness}}\)

Agent responses are classified into five behavioral categories: (a) unaware risky execution, (b) aware but still executes, (c) warns-and-blocks then aborts, (d) warns-and-blocks then completes safely (optimal), (e) silent safe completion. A GPT-5.1-Codex-Mini LLM judge evaluates each run against task artifacts and expected safe behavior specs.

Key Contributions

  • OpenSkillRisk dataset: 263 validated real-world risky skills across 7 attack categories (control-plane hijacking, authority expansion, data harvesting, execution bootstrapping, persistence implantation, outbound exfiltration, external state manipulation) and 9 task domains, split into 139 unconditionally malicious and 124 context-dependent risky skills.
  • Awareness rate metric: A cognition-level metric that separately measures whether agents explicitly warn users, decoupled from whether they succeed in blocking execution.
  • \(F_{\text{safe}}\) composite metric: Harmonic-mean combination of avoidance rate and awareness rate, penalizing systems that are "accidentally safe" or aware-but-non-intervening.
  • Five-category behavioral taxonomy: Granular decomposition of agent failure and success modes beyond binary safe/unsafe.
  • Cross-harness empirical study: First systematic comparison of 3 CLI harnesses under identical risky-skill conditions, showing that harness-level execution control is a distinct safety factor independent of model capability.

Results

  • \(F_{\text{safe}}\) spans 27.86% (Gemini CLI + Gemini 2.5 Pro) to 80.97% (Claude Code + Claude Sonnet 4.6); no system reaches full reliability.
  • Best single-system ASR: 17.11% (Claude Code + Claude Opus 4.6); best overall \(F_{\text{safe}}\): 80.97% (Claude Code + Claude Sonnet 4.6).
  • Context-dependent risks are harder: across 10/15 systems, shifting from unconditionally malicious to contextually risky skills raises average ASR from 22.0% → 35.6% and drops Awareness from 62.9% → 51.4%.
  • Attack category results (averaged across models):
Attack family ASR ↓ Awareness ↑ \(F_{\text{safe}}\) ↑
Outbound Exfiltration 18.54% 70.14% 75.38%
Control-plane Hijacking 49.74% 18.97% 27.55%
Authority Expansion 50.00% 31.58% 38.71%
  • Control-plane hijacking is the hardest category: ASR ~50%, Awareness ~19%, \(F_{\text{safe}}\) 27.55%.
  • Awareness and avoidance are decoupled: Gemini 3.1 Pro reaches 72.62% Awareness yet 26.62% ASR; Claude Haiku 4.5 achieves 20.53% ASR but only 47.91% Awareness.
  • Within Claude Code, upgrading from Haiku 4.5 to Sonnet 4.6 raises \(F_{\text{safe}}\) from 59.78% → 80.97%; within Codex, GPT-5.1-Mini → GPT-5.4 raises it from 44.42% → 78.46%.

Limitations

  • Benchmark covers only English-language CLI skills from two marketplaces (SkillsMP, Skills.rest); non-English and non-CLI ecosystems are unrepresented.
  • 263 skills, though real-world, is a relatively small sample given 175K+ total crawled; stratified sampling may undersample rare attack types.
  • The LLM judge (GPT-5.1-Codex-Mini) introduces evaluator bias; judge agreement is validated in an appendix but not fully addressed in the main results.
  • Sandbox mock stubs eliminate real network side effects, which may underestimate risk severity in live deployments where partial execution can still cause harm.
  • Pre-screening and runtime defenses (e.g., least-privilege enforcement) are not systematically ablated, so the marginal value of each defense layer is not quantified.
  • Skill ecosystems and harness APIs evolve rapidly; the March 2026 snapshot may date quickly.

Relevance to Harnesses / Meta-Harnesses

OpenSkillRisk directly benchmarks CLI agent harnesses (Claude Code, Gemini CLI, Codex CLI) as first-class experimental variables, demonstrating that harness-level execution control contributes to safety outcomes independently of the underlying LLM—exactly the kind of cross-cutting harness evaluation that meta-harness researchers need. The behavioral taxonomy (warn-and-block vs. silent-safe vs. aware-but-proceed) maps onto harness design decisions around permission gates, execution sandboxing, and user-approval checkpoints. The finding that system-level attacks (control-plane hijacking, authority expansion) defeat current harnesses at ~50% ASR identifies a concrete gap in harness-level trust boundaries and tool-call policy enforcement. For meta-harness work concerned with composing or wrapping multiple harnesses, the cross-harness variance in \(F_{\text{safe}}\) (27.86%–80.97%) under identical skill inputs highlights how much safety posture is an emergent property of the harness stack rather than the model alone.