Skip to content

Robust and Diverse Multi-Agent Learning via Rational Policy Gradient

🕒 Published (v1): 2025-11-12 18:34 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RPG (Rational Policy Gradient) addresses self-sabotage in adversarial multi-agent optimization by requiring agents to remain rational (i.e., best-responses to some co-policy) while pursuing adversarial objectives. The method introduces manipulator agents that use opponent-shaping to steer base agents toward adversarial solutions without irrational behavior. This extends adversarial optimization algorithms from zero-sum to general-sum and cooperative settings.

Problem

Adversarial optimization (self-play, adversarial training, adversarial diversity) works well in zero-sum settings but fails in cooperative/general-sum games because adversaries learn to self-sabotage—minimizing a teammate's reward by simply refusing to cooperate or actively blocking the task. Prior fixes (CoMeDi, observation-distribution matching) are partial and environment-specific; no general framework existed to prevent self-sabotage across arbitrary adversarial optimization objectives.

Method

Rationality-preserving Policy Optimization (RPO) formalizes the constraint that each agent's policy must be a best-response to at least one possible co-policy, i.e., remain rational, while simultaneously optimizing an adversarial objective.

Rational Policy Gradient (RPG) solves RPO by introducing a parallel set of manipulator agents (one per base agent): - Base agents train only to maximize reward against their corresponding manipulator (enforcing rationality). - Manipulators optimize the original adversarial objective by taking higher-order gradients through the base agents' update step via opponent shaping (LOLA-style) and Loaded DiCE for unbiased higher-order gradient estimation through autodiff. - Partner-play regularization adds rollouts against other base agents (weighted by small ε) to prevent distributional shift when switching from manipulator-training to evaluation against peers.

The manipulators are discarded after training; only base agents are deployed. RPG is RL-algorithm-agnostic. Five algorithm variants are derived: AP-RPG (adversarial policy), AT-RPG (adversarial training), PAIRED-RPG, PAIRED-Attack-RPG, and AD-RPG (adversarial diversity).

Key Contributions

  • Formal definition of self-sabotage and the RPO framework that prevents it in any adversarial optimization objective via a rationality constraint.
  • RPG algorithm: gradient-based solver for RPO using opponent shaping + Loaded DiCE for higher-order gradients.
  • Five novel RPG-extended adversarial algorithms applicable to general-sum games.
  • Empirical demonstration that AD-RPG produces genuinely diverse (incompatible) policies without sabotage, outperforming AD and CoMeDi in Overcooked and STORM.
  • AT-RPG and PAIRED-RPG train robustified agents that withstand adversarial attacks (AP-RPG, PAIRED-A-RPG) better than non-RPG baselines.
  • AP-RPG finds rational adversarial examples (e.g., counterclockwise movement in Overcooked) rather than trivial sabotage.

Results

  • Adversarial diversity (Overcooked, cramped room): AD-RPG achieves self-play reward of 240 and cross-play reward of 240, vs. CoMeDi (self-play 220, cross-play 2) and AD (self-play 240, cross-play 1.25)—demonstrating sabotage elimination.
  • Robustness (Overcooked cross-play grids): AD-RPG achieves near-perfect cross-play with any partner in forced coordination; SP policies achieve ~0 cross-play with partners from other algorithms.
  • Robustness (Hanabi 3- and 4-color): AD-RPG maintains substantially higher intra-population cross-play rewards than AD and comparable-entropy SP.
  • Adversarial examples (unobserved STORM): AP-RPG achieves reward 0.42–0.96 against various victims; vanilla AP achieves 0.0 across the board due to sabotage. PAIRED-A-RPG/AT-RPG victims score higher against AP-RPG than non-RPG-trained victims, indicating improved robustness.
  • Overcooked AP-RPG: Finds adversarial partner achieving reward 4.6 against a SP victim that normally scores 240 in self-play.

Limitations

  • RPG requires computing higher-order gradients via DiCE, which is computationally expensive compared to first-order methods.
  • The rationality constraint is enforced approximately through gradient-based optimization, not guaranteed exactly.
  • Evaluated in relatively small/structured environments (Overcooked, STORM, simplified Hanabi); scalability to large state spaces or many agents is untested.
  • Population size fixed at 2 for AD-RPG experiments; scaling diversity to larger populations is unaddressed.
  • Partner-play regularization introduces a hyperparameter (ε) that requires tuning.
  • Concurrent work (Wang et al., Chaudhary et al.) addresses overlapping problems; relative advantages in those specific settings are not directly compared.

Relevance to Agentic AI / LLM Agents

Multi-agent robustness and diversity are increasingly critical as LLM-based agents are deployed in collaborative or competitive pipelines—e.g., tool-use agents that must work with unknown partners, multi-agent debate/verification systems, or AI teammates paired with humans. RPG's rationality constraint is directly applicable to agent coordination settings where a "helpful" agent must not degrade to irrational blocking behavior when incentives are misaligned. The AD-RPG diversity mechanism is relevant for generating diverse agent conventions and auto-curricula, which parallels work on self-play-based agent training in LLM contexts. The opponent-shaping foundation also connects to agent modeling and theory-of-mind in agentic systems.