SkillFuzz: Fuzzing Skill Composition for Implicit Intents Discovery in Open Skill Marketplaces¶

🕒 Published (v1): 2026-07-02 15:49 UTC · Source: Arxiv · link

Why this paper was selected

Skill composition fuzzing reveals implicit intents/vulnerabilities in open skill marketplaces

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

SkillFuzz is the first execution-free testing framework for discovering implicit intents—unintended agent objectives that emerge only when multiple individually-benign LLM skills are co-activated in open skill marketplaces. It extracts structured skill contracts, embeds them semantically, and uses contract-guided Monte Carlo Tree Search (MCTS) with plan drift as a differential oracle to efficiently explore the exponential co-activation space. Across a 196-skill benchmark, it discovers >1,000 distinct implicit intents and confirms >80% of highest-risk compositions during execution-time validation.

Problem¶

Skill marketplace operators audit skills in isolation before admission, but composition of benign skills can jointly redirect an LLM agent's belief state toward objectives encoded by neither skill alone—what the paper calls implicit intents. Three obstacles block detection: (1) execution environments are unavailable at admission time; (2) compositional effects exist only in the LLM's internal belief state, invisible to static analysis; (3) the co-activation space $\{0,1\}^{|L|}$ grows exponentially, making exhaustive testing intractable.

Method¶

Formalization. A skill activation is a binary vector $s \in \{0,1\}^n$; the planning agent produces $\pi_s = M_{\text{plan}}(s, \sigma)$. Plan drift is $\delta(s,\sigma) = 1 - \text{sim}(E(\pi_0), E(\pi_s))$, comparing against the skill-free baseline $\pi_0$. The Intent Coverage Quality metric combines severity and novelty: $$\text{ICQ}(s,\sigma) = \delta(s,\sigma) \cdot \nu(s), \quad \nu(s) = \mathbf{1}\!\left[\max_{\psi'\in G(t-1)}\text{sim}(E(\psi(s)),\psi') < \theta\right]$$ The objective is to maximize cumulative ICQ $\sum_{t=1}^{B}\text{ICQ}(s^{(t)},\sigma)$ under query budget $B$.

Step 1 – Skill Contract Extraction & Pruning. An LLM extracts a structured contract $C(s_i) = (P_i, Q_i, M_i, V_i, D_i, A_i, c_i)$ (preconditions, postconditions, modifies-set, invariants, domain scope, abstract action types, extraction confidence) from each instruction document. Contracts are embedded as $v_i = E(C(s_i))$. The candidate set $\Omega_\sigma = \{s_i \in L \mid \text{sim}(v_i, E(\sigma)) \geq \tau_{\text{filter}}\}$ prunes irrelevant skills, and the seed queue $S_0$ prioritizes pairs with high conflict potential (shared modifies entries, mutually exclusive invariants) and low $c_i$.

Step 2 – Differential Activation Search (MCTS). Each MCTS node is a skill activation $s$ with $|s| \leq k_{\max}$; edges are single-skill bit-flips. Selection uses $\text{UCB}(s) = r(s) + c_{\text{UCB}}\sqrt{\ln N(\text{parent}(s))/N(s)}$. When $r(s) > 0$, expansion steers toward the most recently confirmed intent region via $v_{\text{target}} = \bar{v}_s + \alpha(E(\psi) - \bar{v}_s)$, selecting $i^* = \arg\min_{i}\|v_i - v_{\text{target}}\|_2$. Simulation draws $K$ plan samples, scores each with $\delta^{(k)}$ and novelty $\nu^{(k)}$, and backpropagates the mean ICQ as node reward.

Key Contributions¶

Formalizes compositional intent testing as a distinct testing problem: skill compositions are the unit under test, plan-level artifacts serve as the observable surface, and plan drift from a skill-free baseline is the differential oracle.
Proposes SkillFuzz, the first execution-free fuzzing framework for skill marketplaces, combining contract extraction with MCTS-based search guided by the ICQ criterion.
Introduces the ICQ metric that jointly rewards severity ($\delta$) and novelty ($\nu$), preventing budget waste on near-duplicate discoveries.
Demonstrates generality across 8 planning agents (4 open-weight DeepSeek-R1 variants, 4 proprietary GPT-5/4.1 models) on SkillsBench.

Results¶

SkillFuzz discovers >1,000 distinct implicit intents across 10 representative tasks under $B=200$ queries per task.
>80% of the highest-risk flagged skill compositions are confirmed to produce observable implicit intents during execution-time validation (RQ2).
Identifies substantially more high-severity implicit intents ($C_{\geq\delta_{\text{sev}}}$) than alternative search strategies while exploring a fraction of the pairwise interaction space they require (RQ3, $B=1000$).
Open-weight agents: DS-R1-7B yields $\Sigma_C=257$, $\bar{\delta}=0.370$, $\text{Div}=0.443$; larger models expose fewer intents (DS-R1-32B: $\Sigma_C=99$, $\bar{\delta}=0.189$), consistent with greater instruction-following stability.
Proprietary agents: GPT-5-nano yields $\Sigma_C=217$; GPT-4.1-mini yields only $\Sigma_C=40$, $\text{Div}=0.234$, suggesting stronger agentic instruction-following reduces compositional drift.
Every evaluated agent produces a non-zero implicit-intent count, confirming the vulnerability generalizes across model families and providers.

Limitations¶

Relies on the plan-then-act paradigm: agents that do not expose a pre-execution plan artifact cannot be tested with this oracle.
Maximum co-activation depth $k_{\max}=5$ may miss higher-order interactions; cardinality-3+ compositions are underexplored relative to pairs.
Intent extraction uses GPT-4o-mini as a fixed judge, introducing a potential dependency on that model's semantic parsing quality and a self-evaluation asymmetry.
SkillsBench has 196 skills; scalability to real marketplaces with thousands of skills is not empirically validated.
Paper is truncated before full ablation results (RQ3/RQ4) and discussion of false-positive rates on low-severity flagged compositions below the 80% confirmed threshold.

Relevance to Agentic AI / LLM Agents¶

This work directly addresses a critical safety blind spot in the emerging skill-plugin ecosystem for LLM agents: the assumption that per-skill safety audits are sufficient for compositional deployments. By casting skill composition testing as a fuzzing problem and exploiting the plan-then-act architecture as a test oracle, it offers a practical pre-deployment screening tool applicable to any marketplace that surfaces planning artifacts. The finding that compositional vulnerabilities generalize across all tested model families—including strong instruction-following models—underscores that this is an architectural property of multi-skill context fusion, not a quirk of specific models. For researchers tracking agentic AI, SkillFuzz establishes a new testing paradigm alongside existing prompt-injection and red-teaming work, and the ICQ metric provides a reproducible measure of compositional risk coverage.