Skip to content

AgentBreeder: Mitigating the AI Safety Risks of Multi-Agent Scaffolds via Self-Improvement

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AgentBreeder is an evolutionary multi-objective framework that searches the space of Python-coded multi-agent scaffolds to jointly optimize capability and safety. In "blue" mode it achieves a 79.4% average uplift on a safety benchmark while maintaining or improving capability; in "red" mode it automatically surfaces scaffolds that exploit base-LLM safety vulnerabilities while preserving task performance.

Problem

Prior automated agentic system design (ADAS) work optimizes scaffolds purely for capability, ignoring safety. Alignment research has focused almost exclusively on single-agent, unipolar scenarios, leaving the safety impact of multi-agent scaffolding—where LLMs interact in novel multipolar configurations—largely unexplored and unevaluated.

Method

AgentBreeder runs a quality-diversity evolutionary loop over multi-agent scaffolds expressed as Python code. At each generation: (1) scaffolds are evaluated on a capability benchmark (DROP/MMLU/GPQA) and a safety benchmark (SaladData, Constitutional-AI-judged); (2) scaffold embeddings (OpenAI text-embedding-3-small, 12-dim) are agglomeratively clustered (distance threshold 0.7); (3) Pareto-optimal elites from each cluster are selected; (4) a Meta Agent (Claude 3.5 Sonnet) applies mutation (2Ă— weight over crossover) or crossover to produce offspring. Three modes exist: Blue (maximize both capability and safety), Red (maximize capability, minimize safety), and Capable (capability only, ablation). Safety-enhanced vs. capability-enhanced mutation operators are used selectively per mode. Reward-hacking of the safety objective is detected by reporting helpfulness on TruthfulQA.

Key Contributions

  • BlueAgentBreeder: evolutionary defense that discovers scaffolds simultaneously improving safety (up to 110.7%, avg 79.4% uplift on SaladData) and capability (up to 21.0% on GPQA) relative to seed scaffolds.
  • RedAgentBreeder: automated red-teaming that surfaces scaffolds exploiting base-LLM safety weaknesses while maintaining competitive task performance—unsafe scaffolds emerge without explicitly prompting for them, through Pareto optimization alone.
  • Multi-objective ADAS: extends Meta Agent Search (Hu et al.) with a MAP-Elites-style quality-diversity mechanism using dynamic clustering and Pareto-front elite selection.
  • Reward-hacking detection: identifies and filters scaffolds that trivially inflate safety scores (e.g., always refusing) by cross-checking helpfulness on TruthfulQA.
  • Implementation in the Inspect evaluation framework for reproducibility and extensibility.

Results

  • Blue mode (20 generations, run independently per benchmark):
  • SaladData safety: up to 110.7% increase, 79.4% average uplift over best seed scaffolds.
  • GPQA capability: up to 21.0% improvement; MMLU and DROP maintained or marginally improved.
  • HV indicator gains: GPQA 0.219→0.248, MMLU 0.484→0.543, DROP 0.391→0.439.
  • Red mode (10 generations on DROP):
  • Best scaffold: 81.6±4.8 accuracy on inverted safety (1-SaladData), surpassing all seed scaffolds by ≥6.25% with half the generation budget.
  • Capability maintained: 67.7±4.7 F1 on DROP (comparable to seed CoT at 68.2±4.7).
  • HV indicator: 0.573→0.603.
  • Capable mode (ablation): marginal gains over ADAS seeds (e.g., DROP 72.3±3.1 F1); no meaningful safety impact, confirming safety uplift requires explicit multi-objective optimization.
  • Multi-objective outperforms single-objective in capability too, attributed to increased validation signal-to-noise ratio.

Limitations

  • Small population sizes and limited generations (20 blue, 10 red) due to compute cost; only marginal average improvements in absolute benchmark scores.
  • Seed population of only 7 hand-designed scaffolds may constrain diversity of discovered architectures.
  • Black-box evaluation only; agent-agent and agent-tool interactions are not traced, leaving latent risks invisible.
  • Safety benchmark (SaladData + Constitutional-AI judge) may not capture real-world threat diversity; helpfulness proxy (TruthfulQA) is a coarse reward-hacking detector.
  • Performance gains diminish with stronger base LLMs (GPT-4o mini vs. earlier models), possibly due to data contamination and already-strong fine-tuning for reasoning.
  • Experiments use a single safety benchmark; stronger safety claims would require more comprehensive evaluation suites (e.g., AILuminate).

Relevance to Agentic AI / LLM Agents

AgentBreeder directly addresses a critical gap for the agentic AI field: the safety implications of multi-agent scaffolding, which capability-focused ADAS work (e.g., Meta Agent Search) has ignored. The red-teaming result—that unsafe scaffolds emerge automatically through capability-safety Pareto optimization, without explicit adversarial intent—is a significant safety warning for anyone deploying scaffolded LLM agents at scale. The framework is practically applicable as a pre-deployment pipeline for red-teaming new LLMs and for designing production scaffolds that conform to organizational safety policies, directly relevant to the growing ecosystem of web-browsing and tool-using autonomous agents.