Adaptive Adversaries: A Multi-Turn, Multi-LLM Benchmark for LLM Agent Security

🕒 Published (v1): 2026-07-20 15:30 UTC · Source: Arxiv · link

Why this paper was selected

Adaptive multi-turn, multi-LLM security benchmark; dynamic attacker vs. defender evaluation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces a 21-scenario security benchmark for LLM agents in which an adaptive LLM attacker observes prior defender responses and pivots across up to 15 rounds, while each defender response is evaluated as a fresh, memoryless interaction. The key finding is that frontier defenders (Claude Opus 4.6, GPT-5.4, Gemini 2.5 Pro) show markedly different per-scenario weaknesses despite similar aggregate ASR, and that multi-attacker evaluation uncovers 1.4–2.2× more unique successful attacks than any single attacker alone.

Problem

Existing safety benchmarks evaluate defenders against static, pre-collected attack pools — generated before evaluation and reused across all defenders. Even pipelines like PAIR and TAP that adapt attacks during construction release a frozen set post-hoc. No prior benchmark simultaneously features multi-turn dialogues, per-battle live attack regeneration, and a multi-LLM-agent attacker pool, leaving a quadrant of the evaluation design space unoccupied and hiding attacker-model identity as a measurement axis.

Method

Built on the AgentBeats multi-agent platform, each battle pairs one attacker harness with one memoryless defender harness under an orchestrator. The attacker observes all prior defender responses and generates each attack turn live (per-battle regeneration, not from a pool). The defender receives only the current formatted input — no attack metadata, no history — and outputs structured JSON evaluated by a per-scenario check_success function that requires the unsafe outcome to appear in a designated typed field.

Scenarios are authored in three stages: (i) LLM-seeded concept generation across seven attack archetypes, (ii) Python plugin implementation (system prompt, output schema, dialogue template, check_success), and (iii) calibration filtering using a strong (gpt-5.2) vs. weak (gpt-4o-mini) pair across three configs (BAL, SVW, WVS, 10 runs each). A calibration rubric rejects scenarios on four hard flags (BAL_100, SVW_0, NO_DIFF, INVERTED) and three scored dimensions:

Dimension	Target
Difficulty	BAL win rate 30–70%
Sensitivity	SVW − WVS gap > 30 pp
Round depth	Mean rounds-to-win 5–12

The controlled evaluation uses a 3×3 attacker × defender matrix (Claude Opus 4.6, GPT-5.4, Gemini 2.5 Pro) at \(N=5\) per scenario, yielding 945 total battles. Concordance across scenarios is measured with Kendall's \(W\) over per-item defender rankings.

Key Contributions

• A 21-scenario held-out benchmark (+ 10 public development scenarios) filling the multi-turn × per-battle regeneration × multi-LLM-agent quadrant; released with orchestrator, baseline harnesses, multi-attacker CLI, and 945 full transcripts.
• Four design principles: multi-stage calibrated scenario authoring, multi-turn agent-vs-agent battles, per-battle attack regeneration, and structured outcome-based success checks resistant to proxy-metric gaming.
• Empirical demonstration that pooling three frontier attackers yields 1.4–2.2× more unique successful attacks than the best single attacker; attacks show low pairwise cosine similarity (0.02–0.14 in MiniLM-L6-v2 space) to six existing benchmarks.
• Quantitative evidence that frontier-defender weaknesses are not one-dimensional: \(W = 0.19\) (CI [0.02, 0.58]) across 14 informative scenarios, with 13 of 14 discriminative.
• 18,422-battle open competition dataset (gpt-oss-20b backbone) showing that adaptive harnesses generalize to held-out scenarios while fixed-catalog harnesses overfit.

Results

• Aggregate ASR (3×3 frontier, 15 rounds): Opus 5.4%, GPT-5.4 5.4%, Gemini 14.0% (as defenders; Opus is strongest attacker at 11–19% across defenders).
• Round-1-only scoring on the same 945 battles: Opus 0.0%, GPT-5.4 0.0%, Gemini 1.0% — demonstrating that multi-round adaptive evaluation is necessary.
• Scenario-specific gaps: memleak — Opus compromised 60% (CI 36–80%), GPT-5.4 and Gemini each 7% (CI 1–30%); replicated at \(N=20\): Opus 50%, GPT-5.4 10%, Gemini 5%. paperaccept — Gemini 53% (43% at higher N), others ≤2%.
• Multi-attacker diversity: 0 of 46 unique winning attacker turns succeed against all three frontier defenders; attack transfer is asymmetric (GPT-5.4→Opus 25.7%, Opus→GPT-5.4 0%).
• Ablation (adaptive vs. non-adaptive attacker on 3 high-ASR scenarios, \(N=3\), 108 battles): adaptive protocol 30.0% ASR; removing attacker access to prior responses drops to 0/36 (0%); stateful defender drops to 19.4%.
• Open competition (N=42 per submission): submitted attackers reached 9.5–50.0% ASR (median 29.3%), submitted defenders 9.5–78.6% ASR (median 25.0%).
• Concordance: Ours \(W=0.19\); b3 \(W=0.712\); benchmarks with more concentrated signal have higher concordance but hide per-scenario weaknesses.

Limitations

• Small per-cell sample size (\(N=5\)) in the frontier evaluation; large-gap scenarios replicated at \(N=20\) but mid-range scenarios have wide CIs.
• Attacker capability is bounded by LLM safety training — LLM attackers will refuse strategies that human red-teamers might pursue, so the benchmark complements but does not replace human red-teaming.
• Scenario coverage is skewed: 11 of 21 scenarios are indirect injection, 6 are PII/data leak; scenarios are curated for difficulty and sensitivity, so the high informative-scenario rate is partly by construction and not a random-sample coverage estimate.
• Memoryless defender design is a specific threat model (fresh-session adversary); stateful agents may be more or less robust depending on whether history aids or reinforces manipulation.
• Cross-benchmark ASR comparisons are confounded by differing judges, content, and turn structures.

Relevance to Agentic AI / LLM Agents

This benchmark directly addresses a critical gap in evaluating LLM agents under realistic adversarial conditions: real-world agents process external content (tool outputs, documents, web pages) from adversaries who can observe and adapt, yet prior benchmarks used static attack pools that cannot pivot mid-dialogue. The finding that aggregate ASR can mask sharp scenario-specific vulnerabilities — two top-tier defenders tied overall yet each with a 60%+ failure rate on different scenarios — is a strong argument against relying on leaderboard aggregates to assess agent security. The multi-attacker diversity result (each frontier LLM attacker finds distinct successful strategies, with near-zero cross-transfer) implies that single-LLM red-teaming systematically underestimates agent attack surface. For practitioners building LLM agents, the calibrated scenario design methodology and the per-battle live generation protocol offer a reusable template for constructing domain-specific security evaluations beyond the released scenarios.