Benchmarks for Trajectory Safety Evaluation and Diagnosis in OpenClaw and Codex: ATBench-Claw and ATBench-Codex¶

🕒 Published (v1): 2026-04-16 10:45 UTC · Source: Arxiv · link

Why this paper was selected

Trajectory-level safety evaluation and diagnosis benchmark for agent harnesses

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

ATBench-Claw and ATBench-Codex are domain-customized extensions of the ATBench trajectory-level safety benchmark, targeting OpenClaw (tool/skill/session orchestration) and OpenAI Codex (repository/shell/patch execution) settings respectively. The core mechanism is taxonomy-guided customization: adapt the shared three-dimensional Safety Taxonomy (risk source × failure mode × real-world harm) to each execution setting, then feed that specification into the unchanged ATBench data-generation engine. This demonstrates that a single meta-benchmark harness can generate setting-specific safety benchmarks without redesigning the underlying pipeline.

Problem¶

Agent execution settings evolve faster than benchmarks: new tool ecosystems, runtimes, and action surfaces expose new risk regions even when the underlying agent architecture is stable. Existing trajectory-safety benchmarks are static and cannot be systematically extended to new settings (OpenClaw-style stateful orchestration or Codex-style repository/shell execution) without bespoke redesign.

Method¶

ATBench provides two invariant foundations: (1) a structured three-dimensional Safety Taxonomy decomposing agentic risk into risk source, failure mode, and real-world harm; and (2) a data-generation engine combining taxonomy-guided risk sampling, heterogeneous tool sourcing, planner-based trajectory synthesis, paired safe/unsafe construction, and delayed-trigger long-context realization. Domain customization works by writing a new taxonomy specification for each target setting—without modifying the engine—and the engine operationalizes that spec into trajectory data.

ATBench-Claw extends via new taxonomy categories: Sender/Session Identity Ambiguity, Persistent Memory/Session-State Contamination, Skill/Plugin Supply-Chain Compromise, Approval Bypass or Missing Human-in-the-Loop, Cross-Tool Attack Chaining, Cross-Channel/Recipient Misrouting, Unsafe Unattended Automation, plus a new harm axis (Compliance, Legal, and Auditability Harm). Execution context is centered on stateful tool/skill invocations with external side effects.

ATBench-Codex uses a mixed strategy: adds a small number of new categories (Repository Artifact Injection, Dependency/MCP Supply-Chain Compromise, Destructive Workspace Mutation, Unsafe Shell/Script Execution) and reinterprets inherited categories (prompt injection → repository-artifact injection; corrupted tool feedback → misleading build/test output; over-privilege → shell/patch/network scope). Execution context is centered on repositories, patches, sandboxes, and approval policies.

Both benchmarks retain the same trajectory-level task (binary safe/unsafe classification), the same three-way diagnostic decomposition for unsafe trajectories, and the same evaluation schema structure.

Key Contributions¶

Extensibility demonstration: shows that ATBench's taxonomy-plus-engine architecture generalizes to qualitatively different agent execution settings without pipeline changes.
ATBench-Claw taxonomy: formalizes OpenClaw-specific risks (session contamination, approval bypass, cross-tool chaining, routing misrouting, supply-chain compromise) as first-class taxonomy nodes.
ATBench-Codex taxonomy: formalizes Codex-specific risks (repo artifact injection, MCP supply-chain exposure, destructive workspace mutation, unsafe shell execution) and reinterprets inherited nodes under repository/runtime-policy constraints.
Shared evaluation protocol: joint safe/unsafe classification setup with accuracy/F1/recall, plus fine-grained diagnostic slicing by taxonomy leaf, enabling cross-benchmark comparison.
Public datasets: ATBench-Claw and ATBench-Codex released on HuggingFace (AI45Research/ATBench-Claw, AI45Research/ATBench-Codex).

Results¶

The experimental section is truncated in the provided text before numerical results appear. No headline accuracy/F1/recall numbers or baseline comparisons can be reported faithfully from the available text.

Limitations¶

Experimental results are not available in the provided text; claimed evaluation outcomes cannot be verified.
Benchmark trajectories are synthetically generated; coverage of real-world edge cases in OpenClaw and Codex deployments is unverified.
Taxonomy customization is a manual design process; coverage gaps in new risk regions depend on the thoroughness of the domain analysis.
The taxonomy does not include an OpenClaw-specific or Codex-specific harm row for every new risk category, so some new failure modes map to inherited harm rows that may not capture full severity distinctions.
Evaluation is limited to coarse binary classification; fine-grained diagnostic slicing uses the taxonomy as a label space but is not directly optimized for.

Relevance to Harnesses / Meta-Harnesses¶

ATBench is itself a meta-harness for benchmark generation: a parameterized construction pipeline that accepts a taxonomy specification as input and produces a domain-specific benchmark as output, analogous to how a meta-harness accepts a configuration and produces task-specific behavior. This paper makes that meta-harness structure explicit by demonstrating two instantiations (Claw, Codex) produced by swapping the taxonomy spec without touching the pipeline. For researchers tracking harnesses and meta-harnesses, this is a concrete example of specification-driven generation—where the "harness" is the data-generation engine and the "configuration" is the three-dimensional taxonomy—directly paralleling how orchestration harnesses separate policy (what to do) from execution substrate (how to do it). The paper also highlights a key harness design principle: keeping the evaluation task and diagnostic framework fixed across instantiations enables cross-setting comparisons that ad-hoc standalone benchmarks cannot support.