Skip to content

Safety Testing LLM Agents at Scale: From Risk Discovery to Evidence-Grounded Verification

🕒 Published (v1): 2026-07-02 07:08 UTC · Source: Arxiv · link

Why this paper was selected

Scalable safety testing framework for tool-using agents — risk discovery + evidence-grounded verification

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VERA is an end-to-end automated safety testing framework that applies software engineering testing principles to non-deterministic LLM agents through a three-stage pipeline: literature-driven risk taxonomy construction, combinatorial executable test-case generation, and adaptive sandboxed execution with evidence-grounded verification. Evaluated on four production agent frameworks, VERA achieves average attack success rates (ESR) of 93.9% under multi-channel attacks. The authors release VERABench: 1,600 executable safety cases across 124 risk categories.

Problem

Existing agent safety evaluations couple risk definitions, environment implementations, and verification procedures so tightly that extending coverage to new risks or agent architectures requires coordinated changes across multiple system layers. More fundamentally, prior work conflates an unsafe request or stated intent with a realized safety violation—failing to verify whether harmful outcomes were actually produced through executed tool actions with observable environmental effects. Neither prompt-level refusal tests nor trajectory-level benchmarks with hand-coded verifiers scale to the rapidly evolving agentic risk landscape.

Method

VERA structures safety testing as a three-stage self-reinforcing pipeline:

  1. Continuous Risk Exploration. A Summary Agent iteratively processes agent-safety literature to build three orthogonal taxonomies: risk type \(R\) (harmful consequence), attack method \(M\) (induction mechanism), and environment \(\Omega_E\) (tool execution context). Each exploration iteration applies create/update/merge/delete operations to converge the taxonomy rather than growing it monotonically: \((T_{t+1}, Q^{(t+1)}) = \Phi(T_t, Q^{(t)}, P^{(t)})\).

  2. Executable Test-Case Construction. Each safety case is a triple \(\sigma = \langle g, s_0, V_g \rangle\). Safety goals \(g = G(r, m, e; D)\) are produced by composing one leaf from each taxonomy via an LLM goal composer. The initial environment state \(s_0\) is programmatically constructed through service APIs (e.g., seeding a password-bearing email for credential-theft goals) and recorded as a deterministic replay script. The verifier \(V_g\) is a deterministic Python predicate over the post-execution trajectory \(\tau\) and terminal state \(s_T\). Each base case is expanded into benign, single-channel, and multi-channel adversarial variants.

  3. Adaptive Execution and Evidence-Grounded Verification. Target agents run in isolated MCP-mediated sandboxes behind a configurable tool gateway that records both the true result \(f_{i,j}\) and the agent-observed result \(\tilde{f}_{i,j}\). An adaptive Control Agent observes runtime behavior and steers multi-turn interaction via control policy \(\pi_{\text{ctrl}}(\sigma, c_i)\), reformulating requests on refusal and adjusting injection points dynamically. Verification applies a priority chain: \(V_g = V_g^{\text{state}}(s_T) \triangleright V_g^{\text{tool}}(\tau) \triangleright V_g^{\text{resp}}(\tau)\), preferring tamper-resistant environment state over agent self-report. A case is marked \(y=1\) only after the verifier confirms the claim against environmental evidence, eliminating false positives from optimistic self-assessment.

Key Contributions

  • Formal definition of the executable safety case \(\sigma = \langle g, s_0, V_g \rangle\) that instantiates SE testing primitives (test oracles, combinatorial construction, evidence-grounded verification) for non-deterministic agents.
  • The VERA framework: a three-stage pipeline supporting heterogeneous agent frameworks via a unified execution contract and isolated stateful sandboxes.
  • VERABench: 1,600 executable safety cases spanning 124 risk categories across three execution settings (benign, single-channel, multi-channel), with deterministic programmatic verifiers.
  • Empirical safety evaluation of four production agent frameworks (OpenClaw, Hermes, Codex, Claude Code), revealing pervasive and severe vulnerabilities.

Results

  • Average ESR across all risk categories under multi-channel attacks: 93.9%.
  • By risk category (multi-channel, averaged across environments):
  • Integrity violations: 95.3%
  • Cyber attack facilitation: 88.4%
  • Privacy & data leakage: 83.7%
  • System abuse: 82.7%
  • Privilege escalation: 82.4%
  • Malware generation: 81.6%
  • System probing: 81.6%
  • Harmful output: 79.0%
  • Several environment/attack-method combinations reach 100% ESR (e.g., CRM & Service + Instruction Injection; Dev & Data + Roleplay/Persona; Jailbreak across multiple environments).
  • Results are reported across 11 environment groups (Communication, Productivity, Finance, CRM & Service, Dev & Data, Social, Travel, Domain-Specific, OS/Terminal, Web & Storage) and 10+ attack method categories.

Limitations

  • Scope is restricted to inference-time attacks; training-phase attacks (fine-tuning poisoning, backdoor injection) are explicitly out of scope.
  • Risk taxonomy completeness depends on the coverage of academic literature retrieved; operational intelligence sources (CVE, MITRE ATT&CK) are mentioned as future extensions but not fully integrated in this evaluation.
  • Evaluation covers only four agent frameworks; generalization to other architectures is asserted but not empirically demonstrated.
  • The text is truncated before reporting per-framework breakdown or single-channel vs. multi-channel comparative ESR numbers, limiting full reproducibility of the aggregate claim.
  • LLM-generated initializers and verifiers may contain errors (syntax/schema mismatches are handled by regeneration, but correctness of the predicate logic is not independently audited).

Relevance to Agentic AI / LLM Agents

VERA directly addresses the core challenge of rigorously evaluating safety in tool-augmented LLM agents operating autonomously across real external services—a defining characteristic of the agentic paradigm. The framework's evidence-grounded verification (environmental state over self-report) and adaptive multi-turn adversarial execution are essential primitives for any agent safety evaluation methodology that must handle non-deterministic planning and diverse tool ecosystems. The finding that four production-grade frameworks exhibit >93% ESR under multi-channel attacks underscores that current agent architectures lack adequate safety boundaries, making this directly relevant to alignment and deployment work on agentic systems. VERABench also provides a reproducible, extensible benchmark that can track safety regressions as agent capabilities evolve.