Skip to content

Benchmarking Autonomous Agents against Temporal, Spatial, and Semantic Evasions

🕒 Published (v1): 2026-05-21 11:07 UTC · Source: Arxiv · link

Why this paper was selected

Benchmarks agents against temporal, spatial, semantic evasions in persistent systems

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A3S-Bench is a security benchmark of 2,254 multi-turn agent trajectories that stress-tests LLM-based autonomous agents (specifically OpenClaw-class harnesses) against three novel evasion strategies: temporal (cross-turn payload fragmentation), spatial (artifact-embedded injection), and semantic (benign-context concealment). Across 10 LLM backbones and 20 threat scenarios, the three advanced strategies raise the average risk trigger rate from 28.3% (naive injection baseline) to 52.6%, with the most vulnerable configuration reaching 77.7%.

Problem

Existing agent security evaluations target stateless, single-turn, or simulated environments with small datasets (44–291 cases, ≤6 risk categories, ≤6 LLMs), failing to model the persistent-state and multi-turn attack surfaces inherent in real autonomous agent frameworks like OpenClaw. No prior work combines multi-turn injection, diverse injection vectors, joint security-utility scoring, and large-scale evaluation in a single unified framework.

Method

The paper constructs A3S-Bench via a three-stage automated synthesis pipeline:

  1. Seed Generation: A cross-product of 6 real-world usage scenarios (code, browser, file, doc, numbers, OS admin) × 10 risk categories is seeded by a pool of diverse LLMs, producing multi-turn benign trajectories with rich workspaces and inter-turn dependencies.
  2. Seed Curation: Feasibility filtering (shell syntax validation), similarity deduplication, defect repair, and quality scoring ensure diversity and executability.
  3. Payload Injection: Three advanced attack strategies are applied, each exploiting a distinct safety-mechanism weakness:
  4. Cross-turn fragmentation (temporal): decomposes a malicious objective into \(k\) innocuous fragments \(\{\pi_1,\ldots,\pi_k\}\) delivered across turns \(I=\{i_1,\ldots,i_k\}\), satisfying \((\forall l,\, x_{i_l}\oplus\pi_l\notin\mathcal{R}) \wedge \Gamma(\pi_1,\ldots,\pi_k)\in\mathcal{R}\), so per-turn safety filters never trigger but the composed payload is harmful.
  5. Detection-scope evasion (spatial): plants the payload in workspace artifacts (skill scripts, config files, project templates) accessed via tool invocations, placing it entirely outside conversation-channel monitoring.
  6. Benign-context concealment (semantic): embeds a dangerous instruction \(c^*\) among legitimate workflow steps \(u_i=\langle c_1,\ldots,c^*,\ldots\rangle\) so it is semantically indistinguishable from surrounding benign content.

All cases execute against a real OpenClaw instance in containerized isolation, with action-based rubric scoring that jointly measures security (risk trigger rate) and utility (task completion).

Key Contributions

  • A risk taxonomy of 3 classes, 10 categories, and 34 attack techniques covering boundary breach, persistent state corruption, and harmful operations in autonomous agent systems.
  • Three formalized advanced evasion strategies (temporal/spatial/semantic) that exploit architectural properties unique to multi-turn, tool-augmented agents.
  • A3S-Bench: 2,254 multi-turn trajectories (1,512 adversarial, 742 benign), two difficulty levels, six scenarios, the largest and most comprehensive autonomous-agent security benchmark to date.
  • Large-scale evaluation consuming >20 billion tokens across 10 LLM backbones, with supplementary defense experiments showing existing guardrails are insufficient.

Results

  • Baseline naive prompt injection: 28.3% average risk trigger rate across 10 LLMs on OpenClaw.
  • Advanced evasion strategies raise the average to 52.6%; worst-case backbone reaches 77.7%.
  • Multi-turn injection (58.6%) substantially outperforms single-turn injection (34.7%), confirming per-turn safety checks cannot detect fragmented intent.
  • Sandbox escape and information leakage are the most exploitable categories across all backbones.
  • Agent-specific threats (memory tampering, malicious skill injection) trigger at >30% even for the safest proprietary models under advanced attacks.
  • All 10 models fail to detect persistent-state attack chains where an initial memory compromise silently propagates to information leakage via deferred triggers.
  • Existing guardrail models and system-level rule updates provide inadequate mitigation against the three advanced strategies.

Limitations

  • Evaluation is restricted to the OpenClaw agent framework; generalizability to other harnesses (e.g., LangGraph, AutoGen, custom orchestrators) is unverified.
  • The paper text is truncated before full description of the seed curation and defense experiment details.
  • Adversary is restricted to black-box, injection-only access; no white-box or weight-level attacks are considered.
  • Benign utility evaluation methodology and rubric scoring details are only partially described.
  • Dataset generation relies on LLM-based seed synthesis, which may introduce biases or gaps in scenario coverage.

Relevance to Harnesses / Meta-Harnesses

This paper directly targets the security of agent harnesses (OpenClaw-class frameworks that orchestrate tool-invocation chains, persistent memory, and plugin ecosystems), making its findings immediately applicable to anyone building or auditing such harnesses. The automated three-stage synthesis pipeline is itself a meta-harness—it generates, curates, and injects adversarial scenarios programmatically, demonstrating how harness-level automation can be repurposed for systematic security stress-testing. For researchers building multi-agent or agentic meta-harnesses, the taxonomy of persistent-state corruption (memory tampering, config poisoning, malicious skill injection) directly maps to attack surfaces introduced by harness features like shared memory stores and plugin registries. The finding that cross-turn fragmentation defeats per-turn safety filters is especially relevant: meta-harnesses that route sub-agent outputs back into orchestrator context without cross-turn aggregation inherit this vulnerability.