Skip to content

AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AutoRedTeamer is a fully automated, end-to-end red teaming framework for LLMs that combines a dual-agent architecture with a memory-guided attack selection mechanism. A strategy proposer agent continuously discovers and implements new attack vectors by analyzing research literature, while a red teaming agent executes test cases using a learned attack library. It achieves 20% higher ASR on HarmBench vs. prior methods while reducing computational cost by 46%.

Problem

Existing automated red teaming approaches either rely on human-curated test cases (poor scalability), optimize individual attack vectors in isolation (miss synergies), or cannot integrate emerging attacks without manual effort. None maintain a persistent memory of attack performance across sessions or operate end-to-end from high-level risk categories without predefined test prompts.

Method

AutoRedTeamer operates in two interlocked phases:

  1. Strategy Proposer Agent: Queries Semantic Scholar API for recent jailbreaking papers, scores each for novelty/distinctiveness/reported effectiveness, then uses an Attack Designer to implement promising attacks as Python classes inheriting a base interface. Attacks are validated on a HarmBench subset (ASR ≥ 30% threshold) before entering the library. The agent also synthesizes novel attack proposals by recombining principles across papers.

  2. Red Teaming Agent: Comprises four modules — (a) Risk Analyzer decomposes user input (risk category or specific prompt) into testable components; (b) Seed Prompt Generator produces diverse test cases across demographic, technical, and contextual dimensions; (c) Strategy Designer selects attack combinations from the library using the memory system; (d) Attack Judge + Relevance Checker evaluate success and semantic drift.

Memory System: Three-tier architecture — long-term memory stores prior test cases and their attack selections (retrieved via embedding similarity for new cases); attack metrics memory tracks per-attack and per-combination success rates, query costs, and execution counts; short-term memory tracks the current test case trajectory. This guides the Strategy Designer toward historically effective combinations.

Initial library: PAIR, ArtPrompt, HumanJailbreaks, Universal Pliny Prompt; expanded by 8 more attacks during integration (Past Tense, few-shot, mutation variants, logic puzzle, obscure cultural references).

Key Contributions

  • Dual-agent architecture (strategy proposer + red teaming agent) enabling end-to-end automated red teaming from raw risk categories or specific prompts
  • Multi-tier memory system tracking attack combination effectiveness across sessions, enabling learned strategy selection
  • Lifelong attack integration pipeline: automated paper retrieval → scoring → implementation → validation → library update
  • First automated system covering all 314 level-4 AIR taxonomy risk categories without human-curated prompts

Results

  • HarmBench, Llama-3.1-70B: ASR 0.82 vs. PAIR 0.60, TAP 0.60, AutoDAN-Turbo 0.67 (+20% over best baseline); 14 evaluation queries vs. TAP's 762 and AutoDAN-Turbo's 8 + 60k development queries
  • HarmBench, Claude-3.5-Sonnet: ASR 0.28 vs. near-zero for all baselines (PAIR 0.04, TAP 0.05, AutoDAN-Turbo 0.02)
  • HarmBench, Mixtral-8x7B: ASR 0.96, matching AutoDAN-Turbo at ~10× fewer evaluation queries
  • AIR categories (risk category inputs): AutoRedTeamer consistently exceeds AIR-Bench human-curated ASR across all 43 level-3 categories; e.g., Llama-3-Instruct-8B 0.90 vs. 0.21
  • Prompt diversity: Average pairwise cosine similarity 0.45 (AutoRedTeamer) vs. 0.68 (PAIR), approaching human-curated AIR-Bench at 0.38
  • Ablations: Removing memory → ASR drops from 0.82 to 0.12 (random) or 0.43 (fixed); removing Strategy Designer → 0.31; removing Attack Proposer → 0.75

Limitations

  • Attack implementation relies on LLM-based code generation (Claude-3.5-Sonnet), introducing potential biases and implementation errors
  • Strategy proposal may inherit biases from the LLM used for scoring and synthesis
  • Validation threshold (ASR ≥ 30% on a HarmBench subset) may not generalize to novel risk domains
  • Black-box constraint on attack design excludes gradient-based or white-box strategies that can be more potent
  • Evaluation primarily on text-based LLMs; authors note extension to agent-based targets as future work

Relevance to Agentic AI / LLM Agents

AutoRedTeamer is itself a multi-agent system — a red teaming agent orchestrating specialized submodules and a strategy proposer agent with tool use (API queries, code generation, validation) — making it a concrete instantiation of agentic LLM design patterns applied to security evaluation. The memory-guided attack selection mirrors broader agent memory architectures (episodic + semantic memory for adaptive decision-making), and the lifelong integration loop demonstrates how agents can autonomously expand their own capabilities by reading and implementing from research literature. For researchers tracking agentic AI, this work also raises a direct safety concern: as LLM agents become more capable of autonomous red teaming, the attack surface for deployed agentic systems (mentioned as future work) grows correspondingly.