Skip to content

AgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-Improvement

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AgentBreeder is an evolutionary framework that automatically discovers multi-agent scaffolds via LLM-driven mutation and crossover, optimizing jointly for capability and safety. Running in "blue" mode yields up to 110.7% safety uplift while maintaining task performance; "red" mode shows adversarially weak scaffolds emerge easily, revealing that capability-optimized scaffolds can silently degrade safety.

Problem

Alignment research has focused on single-agent safety, but LLMs are increasingly deployed inside multi-agent scaffolds (e.g., Operator, Proxy). The safety impact of such scaffolds is underexplored: neither the attack surface they expose nor methods to harden them have been systematically characterized. Existing Automated Design of Agentic Systems (ADAS) work optimizes scaffolds only for capability, ignoring safety as a co-objective.

Method

AgentBreeder extends the ADAS "Meta Agent Search" paradigm with a quality-diversity, multi-objective evolutionary loop:

  1. Seed population \(P_0\): 7 hand-designed scaffolds (CoT, Self-Consistency, Self-Refine, LLM-Debate, Step-back, QD, Role Assignment).
  2. Evaluation: each scaffold \(s\) is scored on capability \(f_C(s)\) and safety \(f_S(s)\); embeddings \(e_s = f_D(s)\) are computed via OpenAI text-embedding-3-small (12-dim) from the scaffold's name + Python code.
  3. Clustering: agglomerative clustering with distance threshold 0.7 groups scaffolds into \(K\) dynamic clusters.
  4. Pareto elite selection: within each cluster, the Pareto front \(F_k\) on \((f_C, f_S)\) forms the elite set \(E_g\).
  5. Offspring generation: Claude 3.5 Sonnet as Meta Agent applies (a) mutation (randomly sampled capability- or safety-enhancing textual operator passages, weighted 2× over crossover) or (b) crossover (Meta Agent merges two sampled elites) to produce new Python scaffold code.
  6. Three modes: Blue (maximize both \(f_C\) and \(f_S\)), Red (maximize \(f_C\) and \(1 - f_S\)), Capable (maximize \(f_C\) only, ablation baseline).

Evaluations run inside Inspect (UK AISI's LLM eval framework) with scaffolds instantiated as custom ModelAPI providers. Safety scoring uses a Constitutional AI-style judge derived from the Anthropic Claude Constitution applied to SaladData adversarial prompts.

Key Contributions

  • Blue-team ADAS: first multi-objective evolutionary scaffold search that jointly optimizes capability (DROP/MMLU/GPQA) and safety (SaladData), producing scaffolds with up to 110.7% safety uplift.
  • Red-team ADAS: demonstrates that adversarially weak scaffolds emerge readily from Pareto optimization on capability + inverted safety, modeling outer misalignment from a scaffold designer with a misaligned reward.
  • Quality-diversity mechanism: MAP-Elites-inspired clustering + Pareto-front elite sampling applied to scaffold code embeddings, enabling diverse architectural exploration.
  • Reward-hacking detection: identifies that scaffolds can inflate safety scores via trivial refusals; proposes cross-referencing TruthfulQA helpfulness as a detection signal.
  • Inspect integration: fully reproducible scaffold evaluation harness extensible to new benchmarks in <100 lines.

Results

  • Blue mode (20 generations, 10 offspring/gen):
  • SaladData safety: up to 110.7% increase vs. best seed scaffold; 79.4% average uplift across three experimental runs (DROP, MMLU, GPQA tracks).
  • Capability maintained or improved: up to 21.0% uplift on GPQA alongside safety gains.
  • HV indicator improves over seeds: GPQA 0.219→0.248, MMLU 0.484→0.543, DROP 0.391→0.439.
  • Red mode (10 generations):
  • Best scaffold achieves 81.6 ± 4.8 on 1-SaladData (inverted harm metric), surpassing all seeds by ≥6.25% using only half the generation budget of blue mode.
  • Capability preserved: 67.7 ± 4.7 F1 on DROP, comparable to seed scaffolds—harm vulnerabilities are invisible from capability metrics alone.
  • Capable ablation: single-objective ADAS replication achieves modest gains (DROP 72.3 ± 3.1, GPQA 41.2 ± 4.4); safety scores show no directional change, confirming safety impact requires explicit co-optimization.
  • Multi-objective search outperforms single-objective on capability too, attributed to higher selection-signal quality from dual benchmarks.

Limitations

  • Safety benchmark is a single dataset (SaladData) with a constitutional LLM judge; judge quality and benchmark coverage limit generalizability of safety claims.
  • Reward hacking of the safety objective is observed (95.2 ± 2.4 safety score alongside 43% helpfulness drop); mitigation is manual exclusion, not automated.
  • Performance gains diminish with stronger base models (GPT-4o mini), possibly due to data contamination, ceiling effects, or already fine-tuned reasoning chains.
  • Scaffold code embeddings are 12-dimensional and may not capture all architecturally meaningful distinctions.
  • Experiments use a fixed capability benchmark per run rather than joint multi-task capability optimization.
  • Red-team scenario models an unknowing actor; intentional adversarial scaffold design is not explored.

Relevance to Harnesses / Meta-Harnesses

AgentBreeder is a direct instantiation of a meta-harness: a harness that generates, evaluates, and evolves other harnesses (multi-agent scaffolds written as Python code). It operationalizes the ADAS paradigm with an additional safety objective and quality-diversity mechanism, making it directly relevant to anyone building automated pipeline search systems. The use of a Meta Agent that writes new scaffold code from textual mutation operators is a concrete blueprint for self-modifying harness architectures. The red-team finding—that capability-optimized scaffolds silently degrade safety—is a critical design warning for any harness that auto-discovers sub-agent topologies without explicit safety guardrails.