PLAGUE: Plug-and-play framework for Lifelong Adaptive Generation of Multi-turn Exploits

🕒 Published (v1): 2025-10-20 17:37 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PLAGUE is a three-phase, plug-and-play meta-framework for generating multi-turn LLM jailbreaks, decomposing attacks into Planner, Primer, and Finisher stages where existing attack methods (GOAT, Crescendo, ActorBreaker) drop in as interchangeable modules. A lifelong-learning memory bank of successful strategies, indexed by goal embedding, is retrieved via cosine similarity to initialize future attacks. It achieves state-of-the-art attack success rates, including 81.4% (SRE) on OpenAI o3 and 67.3% on Claude Opus 4.1.

Problem

Multi-turn jailbreaks are orders of magnitude more effective than single-turn attacks, yet existing multi-turn methods (Crescendo, GOAT, ActorBreaker, RACE) each address only a subset of the required capabilities — plan quality, context progression, adaptive feedback, and strategic diversity. None incorporate lifelong learning; they plateau or drift semantically across multi-goal attack runs. There is no formal framework that disentangles and recombines these components.

Method

PLAGUE orchestrates three sequential phases over a black-box target LLM \(T\):

1. Planner: Given goal \(p_i\), retrieves up to two past successful strategies from a vector memory bank \(R^{+}\) using cosine similarity between goal embeddings (Qwen3-Embedding-0.6B), then prompts the Attacker LLM \(A\) (Deepseek-R1) with these in-context examples to produce an \(n\)-step attack plan.

2. Primer: Executes steps \(1, \ldots, n-1\) of the plan via \(A\), anchoring each round to its plan step rather than the raw goal to prevent semantic drift. A Rubric Scorer \(R\) (Qwen3-235B-A22B) rates each turn on Compliance, Practicality, Level of Detail, and Relevance (10-point scale). Scores below 7/10 trigger backtracking — the turn is removed from \(H_T\) but retained in \(H_A\) — and a reflection-informed retry.

3. Finisher: With context frozen from Primer, the Attacker targets the original \(p_i\) directly. Refusals trigger backtracking at a relaxed threshold (3/10); success requires score \(>8/10\). Any of GOAT, Crescendo, or custom logic can substitute as the Finisher module.

Lifelong Learning: Successful strategy names, definitions, and exact question sequences are stored in \(R^{+}\) indexed by goal embedding. Subsequent runs retrieve these as ICL examples, enabling cross-goal knowledge transfer without gradient updates.

\[\text{ASR}(J) = \frac{1}{P} \sum_{i=1}^{P} J(p_i, \text{MT}_i)\]

Results are reported as ASR@2 (best of \(K=2\) independent runs), averaged over three repetitions.

Key Contributions

• A modular three-phase (Planner → Primer → Finisher) meta-framework for multi-turn jailbreaks where each phase is independently swappable.
• First multi-turn jailbreak system with a lifelong-learning component: embedding-indexed vector memory \(R^{+}\) accumulates successful strategies across attack objectives and retrieves them for future goals.
• Rubric-based intermediate scorer providing fine-grained, four-dimensional feedback within the attack loop rather than only terminal evaluation.
• Empirical demonstration that plug-and-play composition improves both ASR and diversity simultaneously — e.g., swapping in ActorBreaker's planner raises diversity 15% with negligible ASR drop; swapping in Crescendo as Finisher yields +40.2% on Claude Opus 4.1.

Results

• Deepseek-R1, GPT-4o, Llama 3.3-70B: up to 97.8% SRE-ASR within six turns.
• OpenAI o3: 81.4% SRE-ASR; +32.14% over prior SOTA baseline.
• Claude Opus 4.1: 67.3% SRE-ASR; +40.2% over prior SOTA baseline.
• Overall: >30% ASR improvement across leading models at equal or lower query budget compared to Crescendo, GOAT, and ActorBreaker.
• ActorBreaker plateaus at ~60% ASR on most models; Crescendo reaches as low as 37.4% on o3; PLAGUE exceeds both by 30–90% in head-to-head evaluation.
• Evaluated on HarmBench 200-sample standard set with both SRE (graded) and binary-ASR metrics.

Limitations

• Requires multiple LLM calls per turn (Attacker, Rubric Scorer, Summarizer, Evaluator), increasing compute and API cost relative to single-component baselines.
• Lifelong memory bootstrap requires human-seeded initial strategies (two from Crescendo); cold-start diversity is limited until the memory populates.
• Strategy retrieval is goal-to-goal cosine similarity; retrieval degenerates when the library is sparse for a given semantic neighborhood.
• ASR@2 metric with \(K=2\) runs inflates reported numbers relative to single-run baselines, making direct comparison with some prior work approximate.
• Evaluated exclusively on black-box API access; open-weight, locally hosted models and adaptive defenses are not the primary evaluation target.

Relevance to Harnesses / Meta-Harnesses

PLAGUE is architecturally a meta-harness: it defines a compositional scaffold (Planner/Primer/Finisher) into which independent attack modules slot as interchangeable components, exactly the plug-and-play pattern that characterizes meta-harness design. The lifelong-learning memory layer — retrieving successful execution traces to seed future runs — is a direct instance of cross-run knowledge accumulation, a capability that distinguishes harnesses from one-shot scripts. The explicit scoring/feedback loop with backtracking mirrors the monitor-retry control flow common in agentic harnesses. For researchers building or studying AI agent harnesses, PLAGUE demonstrates how decomposing a complex multi-step task into phases with clean interfaces, swappable modules, and persistent memory improves both effectiveness and modularity in an adversarial setting.