Skip to content

CoP: Agentic Red-teaming for Large Language Models using Composition of Principles

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CoP (Composition-of-Principles) is an agentic red-teaming framework that automates LLM jailbreak generation by having a Red-Teaming Agent LLM dynamically select and compose human-provided attack principles. It achieves state-of-the-art single-turn attack success rates with dramatically fewer queries to the target model, exposing vulnerabilities in highly aligned models that evade all prior methods.

Problem

Existing jailbreak methods suffer from at least one of: (1) requiring white-box gradient access (GCG), (2) unguided search with large query budgets (PAIR, TAP), (3) resource-intensive per-model strategy library pretraining (AutoDAN-Turbo), or (4) heavy multi-turn interaction (X-Teaming). On strongly aligned models like Llama-2, the best prior single-turn methods top out around 35–47% ASR, leaving a false impression of safety.

Method

CoP uses three LLMs in a pipeline: a Red-Teaming Agent (default: Grok-2), a Target LLM, and a Judge LLM (default: GPT-4). Given a harmful query, the Red-Teaming Agent first generates an initial seed prompt \(P_\text{init}\) to avoid direct refusal, then iteratively selects and composes principles from a human-curated Principle Inventory (7 principles including Generate, Expand, Phrase Insertion, Style Change) using in-context demonstrations. The resulting composite strategy (e.g., Expand ⊕ Phrase Insertion) is applied to produce a jailbreak prompt. The Judge LLM scores both jailbreak effectiveness (1–10) and semantic similarity to the original query (1–10); prompts that drift below similarity threshold \(\tau\) are discarded. The best-scoring prompt is updated iteratively for up to \(N\) attempts, with early stopping when the jailbreak score exceeds threshold \(\eta\). No gradient access or target-model pretraining is required (black-box only).

Key Contributions

  • Unified, extensible Composition-of-Principles agentic framework that orchestrates human-authored attack principles without retraining per target model.
  • State-of-the-art single-turn ASR across open-source (71–77% on Llama/Gemma families) and proprietary LLMs (88.75% GPT-4-Turbo, 78% Gemini Pro 1.5, 60% OpenAI O1, 38% Claude-3.5 Sonnet — 19× improvement over prior baselines on the last).
  • Up to 17.2× fewer queries to target LLM versus TAP; CoP needs ~1.4–1.5 queries on average for successful attacks on Gemini and GPT-4.
  • Transparent, inspectable attack strategies: identifies that Expand is the most effective single principle (12% of successes), with Expand ⊕ Phrase Insertion (9.8%) and Expand ⊕ Style Change (6.0%) as top compositions.

Results

  • Llama-2-70B-Chat: 72.5% ASR vs. 6.2–47.2% for all baselines (HarmBench, 400 queries).
  • Llama-3-8B-Instruct-RR (Circuit Breaker defense): 52% ASR vs. 26% best baseline (TAP).
  • Llama-2-7B-Chat + Llama-Guard-3: 34% ASR vs. 12% best baseline (TAP).
  • GPT-4-Turbo-1106: 88.75% ASR, 1.512 avg. queries (vs. PAIR: 31.6% / 12.11 queries; TAP: 35.8% / 26.08; AutoDAN-Turbo: 88.5% / 5.63).
  • Gemini Pro 1.5: 78.0% ASR, 1.357 avg. queries (vs. AutoDAN-Turbo best baseline: 66.3% / 2.76).
  • OpenAI O1: 60% ASR on 50 queries (10× over GCG-T baseline).
  • Claude-3.5 Sonnet: 38% ASR on 50 queries (19× over baseline methods near 0%).
  • vs. multi-turn X-Teaming: CoP single-turn achieves 64% ASR on Llama-2-7B; X-Teaming requires 5 turns to match this.

Limitations

  • Principles are human-authored; expanding coverage requires manual domain expertise.
  • Evaluated only on HarmBench (400 queries) and JailbreakBench; generalization to other harm taxonomies is not fully characterized.
  • The Red-Teaming Agent itself (Grok-2) occasionally triggers "Direct Refusal" on 16/400 queries, requiring special seed-prompt handling.
  • Query efficiency is measured only over successful attacks; failed-attempt cost is not included in reported numbers, potentially understating true overhead.
  • No defense recommendations are proposed; the paper is offense-only.
  • O1 and Claude-3.5 Sonnet results use only 50 sampled queries, limiting statistical confidence.

Relevance to Agentic AI / LLM Agents

CoP is a direct demonstration that agentic orchestration — a planning LLM dynamically composing tools (here, jailbreak principles) with feedback-driven iteration — outperforms static or randomly-guided methods by a large margin, which is a core capability pattern in agentic AI. The framework illustrates both the power and the dual-use risk of agentic LLM design: the same reasoning-and-composition loop that makes agents effective at complex tasks also makes them effective adversaries against safety guardrails. For researchers building or securing agentic systems, CoP highlights that single-model safety alignment is insufficient against an adversarial agent that can iteratively probe and compose attacks, motivating work on agent-aware defenses and red-teaming infrastructure.