Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification¶

🕒 Published (v1): 2026-07-02 07:08 UTC · Source: Arxiv · link

Why this paper was selected

Scale-up safety-testing pipeline for tool-using agents; risk-discovery to evidence-grounded verification

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

VERA is an end-to-end automated safety testing framework for LLM agents that converts published literature into executable safety cases via a three-stage, self-reinforcing pipeline: literature-driven taxonomy construction, combinatorial test-case generation, and sandboxed adaptive execution with evidence-grounded verification. Applied to four production agent frameworks, it achieves average attack success rates (ESR) of 93.9% under multi-channel threat conditions. It releases VERABench — 1,600 executable safety cases across 124 risk categories — as a reusable evaluation artifact.

Problem¶

Existing agent safety evaluations either conflate stated intent with realized harm (missing whether harmful actions were actually executed and observable) or tightly couple risk definitions, environment implementations, and verification logic, making them brittle and expensive to extend as agent capabilities and tool ecosystems evolve. No existing framework continuously discovers emerging risks and converts them into deterministically verifiable, runtime-grounded test cases at scale.

Method¶

VERA instantiates software-engineering testing principles (test oracles, combinatorial construction, evidence-based verification) for non-deterministic agents through three stages:

Stage 1 — Continuous Risk Exploration. A Summary Agent iteratively builds and consolidates three orthogonal taxonomies — risk category $R$, attack method $M$, execution environment $\Omega_E$ — from retrieved literature. The update operator $\Phi$ applies four operations per retrieved document (create, update, merge, delete) to each taxonomy tree, preventing monotonic growth and keeping the taxonomy convergent. Query frontiers $Q^{(t+1)}$ are generated from sparsely populated branches to guide subsequent retrieval.

Stage 2 — Executable Test Case Construction. Each leaf triple $(r, m, e)$ from the three taxonomies is mapped to a concrete safety goal $g = G(r, m, e; D)$ via an LLM goal composer with typed schema validation. Each goal is compiled into a safety case $\sigma = \langle g, s_0, V_g \rangle$: $s_0$ is programmatically constructed via service APIs (e.g., a password-bearing email for a credential-theft goal) and recorded as a deterministic replay script; $V_g$ is a deterministic Python verifier over the post-execution trajectory $\tau$ and terminal environment state $s_T$. Three controlled variants are produced per base case: benign, single-channel adversarial, and multi-channel adversarial.

Stage 3 — Adaptive Execution and Evidence-Grounded Verification. Heterogeneous agents are run in isolated sandboxes behind a unified MCP-based tool gateway that records both original tool results $f_{i,j}$ and the potentially transformed observations $\tilde{f}_{i,j}$ returned to the agent. A Control Agent steers multi-turn interaction via an observe–adapt–act loop, reformulating requests on refusal and adjusting injection points based on runtime observations. Verification uses a manipulation-resistant priority chain: $$V_g(\tau, s_T) = V_g^{\text{state}}(s_T) \rhd V_g^{\text{tool}}(\tau) \rhd V_g^{\text{resp}}(\tau)$$ with environment state taking precedence over tool-call records, which take precedence over agent-reported responses. Self-reported success by the Control Agent requires confirmation from $V_g$ before $y=1$ is assigned.

Key Contributions¶

Formal definition of the executable safety case $\sigma = \langle g, s_0, V_g \rangle$ as the primitive for agent safety testing, grounded in observable artifacts rather than model self-report.
VERA framework: three-stage pipeline for autonomous risk discovery, combinatorial test-case generation, and runtime-adaptive sandboxed execution of heterogeneous agent frameworks.
VERABench: 1,600 executable safety cases spanning 124 risk categories across three execution settings (benign / single-channel / multi-channel), with deterministic Python verifiers.
Empirical evaluation of four production agent frameworks (OpenClaw, Hermes, Codex, Claude Code) revealing substantial safety weaknesses.

Results¶

Average ESR (execution success rate for adversary) reaches 93.9% under multi-channel attacks across all frameworks.
By risk category (multi-channel, Table I):
Integrity violations: 95.3% avg ESR across environments
Cyber Attack: 88.4% avg ESR
Sys Probing: 81.6% avg ESR
Privacy & Data: 83.7% avg ESR
Malware Gen: 81.6% avg ESR
By environment group (Table II), CRM & Service and Dev & Data reach 100% ESR under several attack methods (Jailbreak, Instruction Injection, Roleplay).
No quantitative baseline comparison to prior frameworks is reported in the provided text; the four evaluated frameworks serve as primary subjects, not a comparative ablation.

Limitations¶

The text is truncated before Table II is complete; full per-method ESR figures across all environments are not available in the excerpt.
No ablation isolating the contribution of each VERA stage (taxonomy quality, combinatorial generation, adaptive driver) is reported in the provided text.
Scope is limited to inference-time attacks; training-phase attacks (fine-tuning poisoning, backdoor injection) are explicitly out of scope.
Taxonomy coverage depends on retrieval budget and indexed literature; risks undocumented in public literature may be missed until operational feeds (CVE, MITRE ATT&CK) are integrated.
Verifier generation relies on an LLM; syntax or schema errors trigger regeneration but correctness of the verifier logic itself is not formally verified.
No evaluation of false-negative rate (cases where the violation occurred but $V_g$ returned 0).

Relevance to Harnesses / Meta-Harnesses¶

VERA is architecturally a meta-harness: it orchestrates a heterogeneous ensemble of specialized sub-agents (Summary Agent for taxonomy building, goal composer, LLM initializer, Control Agent, verifier generator) within a deterministic outer loop that sequences discovery, generation, and execution phases. The configurable MCP-based tool gateway and per-agent adapters instantiate the harness pattern of wrapping arbitrary upstream systems behind a unified execution contract — exactly the interface abstraction central to meta-harness design. The three-stage, self-reinforcing loop (verified executions feed back into risk exploration) demonstrates how harnesses can achieve closed-loop self-improvement without manual intervention. For researchers tracking harnesses, VERA operationalizes how to compose a multi-agent harness where subagent outputs (taxonomy nodes, safety cases, execution trajectories) become structured data piped to downstream subagents rather than free text.