Skip to content

Empirical Study on Robustness and Resilience in Cooperative Multi-Agent Reinforcement Learning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper presents a large-scale empirical study (82,620 experiments) evaluating cooperation, robustness, and resilience in cooperative MARL across 4 real-world environments, 13 uncertainty types, and 15 hyperparameters. It formalizes the distinction between robustness (stability under persistent perturbation) and resilience (recovery from disruption), finding that these properties do not generalize across uncertainty modalities and that hyperparameter choices often matter more than algorithm selection.

Problem

MARL systems are typically tuned for cooperative performance in idealized simulations, but policies optimized for cooperation frequently fail under real-world uncertainties. The field conflates robustness and resilience, lacks rigorous evaluation across diverse uncertainty types, and ignores the outsized role of hyperparameters in determining trustworthiness under deployment.

Method

The study follows a one-factor-at-a-time design: all MARL models (MADDPG, MAPPO, HAPPO) are first trained with default hyperparameters, then each of 15 hyperparameters is varied independently across 18 tasks in 4 real-world environments (DexHand, Quadrotor, Traffic, Voltage). Robustness is measured as expected return under 13 active uncertainties (Gaussian/greedy/optimal observation noise applied to single or all agents; random/greedy/optimal action perturbations; environmental parameter shifts). Resilience is measured by initializing a new episode from the perturbed state reached after uncertainty exposure and evaluating recovery reward. Pearson correlation and two-way ANOVA (p < 0.001) are used to attribute variance to hyperparameters vs. algorithms.

Key Contributions

  • Formal separation of robustness and resilience in MARL, with distinct metrics grounded in control theory and ecology.
  • First large-scale empirical study (82,620 runs, ~230K GPU-hours on GTX 4090) covering the three-way interaction of cooperation Ă— robustness Ă— resilience.
  • Finding that cooperation–robustness/resilience correlation degrades linearly with attack severity (r = 0.85 and r = 0.76 respectively, p < 0.001).
  • Demonstration that robustness does not generalize across uncertainty modalities (obs/act/env) or agent scopes (single vs. all), confirmed by one-way ANOVA.
  • Hyperparameter prescriptions: early stopping, critic LR > actor LR, and Leaky ReLU consistently improve all three metrics; GAE, PopArt, and parameter sharing consistently hurt them in real-world settings.
  • Averaged improvements from hyperparameter optimization alone: +52.60% cooperation, +34.78% robustness, +60.34% resilience; gains transfer to robust MARL methods (+89.43% / +65.83% / +82.96%).

Results

  • Cooperation–robustness correlation weakens with attack severity: Pearson r = 0.85 (p = 1.03e-10) between performance degradation percentage and cooperation–robustness correlation; r = 0.76 (p = 8.95e-8) for cooperation–resilience.
  • Cross-modality non-generalization: ANOVA confirms obs/act/env robustness are statistically distinct (F(2,153) = 9.53, p < 0.001 for robustness; F(2,153) = 14.51, p < 0.001 for resilience).
  • Single-agent vs. all-agent non-generalization: F(1,142) = 4.36 (p < 0.05) robustness; F(1,142) = 7.00 (p < 0.05) resilience.
  • Hyperparameters outperform algorithm choice in 9/18 tasks by two-way ANOVA (p < 0.001).
  • Early stopping significantly outperforms final-timestep saving: t(161) = 6.16, p < 0.001.
  • Critic LR > actor LR: t(161) = 10.02, p < 0.001 improvement.
  • Leaky ReLU vs. ReLU: t(161) = 6.31, p < 0.001 improvement.
  • GAE hurts: t(107) = 7.44, p < 0.001; PopArt hurts: t(107) = 4.84, p < 0.001 in real-world environments (limited to MuJoCo gains).
  • Parameter sharing hurts in heterogeneous real-world tasks: t(161) = 7.01, p < 0.001.
  • Algorithm differences: MADDPG superior under action uncertainty; MAPPO/HAPPO superior under observation uncertainty.

Limitations

  • Study is empirical rather than theoretical; findings reflect aggregate trends and may not hold in every individual task.
  • Only three algorithms (MADDPG, MAPPO, HAPPO) are evaluated; newer MARL methods (e.g., QPLEX, CDS) are not included.
  • Hyperparameters are varied one at a time; interaction effects between hyperparameters are not systematically characterized.
  • Environmental uncertainty is limited to a single worst-case rollout metric over parameter perturbations; more structured adversarial environment shifts are not explored.
  • Resilience metric depends on the post-perturbation state distribution, which is implementation-specific and may not transfer across simulators.

Relevance to Agentic AI / LLM Agents

While focused on classical MARL rather than LLM-based agents, this work is directly relevant to any multi-agent system deployed under real-world uncertainty—including tool-using or action-taking LLM agents that must maintain cooperation under noisy observations or partial failures. The formal distinction between robustness and resilience (withstanding vs. recovering from disruption) is a conceptual gap that equally afflicts LLM agent frameworks, which rarely evaluate recovery from mid-task failures. The finding that hyperparameter choices dominate algorithm choice is a practically important calibration for researchers building multi-agent LLM systems, where training and inference configurations are often underspecified relative to architectural choices.