Skip to content

Robust and Diverse Multi-Agent Learning via Rational Policy Gradient

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Self-sabotage is a critical failure mode when adversarial optimization is naively applied to cooperative/general-sum multi-agent settings: adversaries learn to simply block task completion rather than find genuine policy flaws. This paper introduces Rationality-preserving Policy Optimization (RPO) and its gradient-based solver Rational Policy Gradient (RPG), which prevent self-sabotage by constraining adversarial agents to remain rational (i.e., best-responding to some valid co-policy). RPG extends five existing adversarial algorithms to cooperative settings, achieving genuine diversity, robustness, and meaningful adversarial examples.

Problem

Adversarial optimization—effective for robustness and diversity in zero-sum MARL—catastrophically fails in cooperative/general-sum settings because adversaries are incentivized to minimize teammates' rewards, leading them to self-sabotage (refuse collaboration, block task completion) rather than expose genuine policy weaknesses. Existing fixes (ADVERSITY, CoMeDi) address self-sabotage only partially and fail across environments.

Method

RPO formalizes a constrained adversarial optimization: each adversarial agent must optimize its adversarial objective subject to its policy being a best-response to at least one possible co-policy (i.e., remaining rational). RPG solves this by introducing manipulator agents—one per base agent—where: - Base agents train solely to maximize reward against their corresponding manipulator (enforcing rationality). - Manipulators use opponent shaping (higher-order gradients via Loaded DiCE) to steer base agents toward policies that optimize the adversarial objective in the original game.

Manipulators are discarded post-training. Partner-play regularization (mixing manipulator rollouts with base-agent rollouts at small weight ε) prevents distribution shift at evaluation. The algorithm is RL-algorithm-agnostic. Five RPG variants are derived: AP-RPG (adversarial policy), AT-RPG (adversarial training), PAIRED-RPG, PAIRED-Attack-RPG, and XPD-RPG (cross-play diversity).

Key Contributions

  • Formal definition of self-sabotage and the RPO framework that eliminates it for any adversarial optimization objective via a rationality constraint.
  • RPG algorithm: gradient-based solver for RPO using opponent shaping with higher-order gradients (Loaded DiCE).
  • Five novel RPG-extended adversarial algorithms applicable to general-sum settings.
  • Demonstration that XPD-RPG fully resolves the open problem of self-sabotage in cross-play diversity (previously unsolved by CoMeDi/ADVERSITY/LIPO).

Results

  • Diversity (Overcooked, cramped room): XPD-RPG achieves cross-play reward of 240 (matching self-play); CoMeDi achieves 2, XPD achieves 1.25—both via sabotage.
  • Robustness (cross-play grids): XPD-RPG maintains high cross-play reward across all partner pairings in forced coordination, coordination ring, counter circuit, and Hanabi (3- and 4-color). SP policies achieve near-zero cross-play with out-of-population partners.
  • Adversarial examples (Overcooked): AP-RPG reduces a self-play victim's reward from 240 to 4.6 while finding a rational (non-sabotaging) adversarial strategy (counterclockwise movement incompatible with victim's clockwise assumption).
  • Rational adversarial attacks (unobserved STORM): AP-RPG and PAIRED-A-RPG expose weaknesses in all victim policies; standard AP achieves only 0 reward (self-sabotage). AT-RPG and PAIRED-RPG victims achieve higher robustness scores against these attacks.
  • Self-sabotage eliminated across all environments and all four RPG algorithm variants; baseline adversarial algorithms (AT, PAIRED, XPD) all fail in training due to sabotage.

Limitations

  • RPG requires computing expensive higher-order gradients (opponent shaping); the paper identifies gradient-free opponent shaping as future work but does not resolve the efficiency cost.
  • Evaluation limited to relatively small cooperative environments (Overcooked layouts, simplified Hanabi, STORM matrix games); scalability to larger action/state spaces is unvalidated.
  • Policies lack history-dependence in Hanabi experiments, which may limit generalization of robustness results to full-complexity settings.
  • RPG introduces additional agents (manipulators) and rollout requirements per base agent, increasing computational overhead.
  • Partner-play regularization introduces a hyperparameter (ε) that may require tuning per environment.

Relevance to Agentic AI / LLM Agents

Robustness to novel partners and the ability to discover diverse behavioral strategies are core requirements for deployed LLM agents operating in multi-agent pipelines or alongside human collaborators. The RPO/RPG framework directly addresses zero-shot coordination and ad-hoc teamwork—settings increasingly relevant as LLM agents are composed into teams with heterogeneous, unknown co-agents. The insight that rationality constraints must be enforced to prevent adversarial agents from trivially exploiting cooperative objectives has direct implications for multi-agent safety and alignment: any adversarial probing mechanism for LLM agent policies (e.g., red-teaming in collaborative settings) must similarly avoid eliciting irrational refusal-to-cooperate as a trivial "attack." The opponent-shaping mechanism underlying RPG also connects to emerging work on agent-to-agent influence and negotiation in LLM multi-agent systems.