ComaDICE: Offline Cooperative Multi-Agent Reinforcement Learning with Stationary Distribution Shift Regularization

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ComaDICE extends the DICE (Distribution Correction Estimation) framework to offline cooperative multi-agent RL by regularizing the joint stationary distribution rather than Q-functions or policies. It introduces a novel value decomposition that factorizes both global value functions and advantage functions through a mixing network, enabling decentralized execution while preserving global optimality. ComaDICE achieves state-of-the-art results on SMACv1, SMACv2, and Multi-Agent MuJoCo benchmarks.

Problem

Offline MARL suffers from distributional shift between the behavior policy (which generated the dataset) and the learned policy. The joint state-action space grows exponentially with agent count, making OOD state-action pairs far more prevalent than in single-agent offline RL. Prior offline MARL methods impose regularization at the Q-function or policy level, which only partially addresses OOD states and actions. No prior work applied stationary-distribution-level (occupancy measure) regularization under the CTDE framework for cooperative MARL.

Method

ComaDICE optimizes a DICE objective that penalizes f-divergence between the occupancy measure of the learning joint policy and the behavior policy:

max E[r(s,a)] − α · Df(ρ^π_tot ∥ ρ^μ_tot)

Under CTDE, the global value function ν^tot(s) and global advantage function A^tot_ν(s,a) are both decomposed via a shared mixing network M^s_θ (a linear combination with non-negative weights) over per-agent local value functions νi(si) and Q-functions qi(si, ai). The paper proves (Theorem 4.2) that the DICE objective is convex in the local values when the mixing network uses non-negative weights and convex activations. The inner maximization over the occupancy ratio w^tot has a closed-form solution w^tot*(s,a) = max{0, f'^{-1}(A^tot_ν / α)}, reducing the minimax to a single minimization. Local policies are extracted via per-agent weighted behavioral cloning using the globally-computed w^tot*, and Proposition 4.3 proves that the product of local WBC-optimal policies equals the globally WBC-optimal joint policy.

Key Contributions

• First application of stationary distribution regularization (DICE) to offline cooperative MARL under the CTDE framework.
• Novel factorization of both ν^tot (value) and A^tot_ν (advantage) through a shared mixing network, rather than Q-functions as in prior MARL work.
• Theorem 4.2: convexity of the DICE objective in local values under non-negative-weight, convex-activation mixing networks.
• Proposition 4.3: formal guarantee that per-agent local WBC optimality implies global WBC optimality.
• First offline MARL evaluation on SMACv2; new SMACv2 offline dataset contributed.

Results

• SMACv1: ComaDICE achieves highest win rates in 10/12 task-quality combinations; e.g., corridor-good: 48.8% vs. OMIGA 41.9%, AlberDICE 43.1%; 2c vs 64zg-good: 55.0% vs. CQL 44.4%, AlberDICE 42.2%.
• SMACv2: ComaDICE leads in 13/15 task instances across Protoss/Terran/Zerg; e.g., Protoss 5v5: 46.2% vs. BC 36.9%, OMIGA 33.1%.
• MaMuJoCo Hopper-expert: 2827.7 ± 62.9 vs. OMIGA 859.6 ± 709.5 and AlberDICE 844.6 ± 556.5 (>3× improvement).
• MaMuJoCo HalfCheetah-expert: 4082.9 ± 45.7 vs. OMIGA 3383.6 ± 552.7.
• Optimal α ≈ 10 across environments; 1-layer mixing network outperforms 2-layer in offline settings.

Limitations

• Restricted to fully cooperative settings; cooperative-competitive scenarios not addressed.
• Performance heavily dependent on behavior policy quality; biased or low-quality offline data degrades results.
• Requires large datasets; sample efficiency is not improved over baselines.
• Global state is approximated by joint observations in partial observability settings, which may introduce inaccuracies.

Relevance to Agentic AI / LLM Agents

Offline MARL is foundational for training multi-agent systems from logged interaction data—directly applicable to scenarios where deploying LLM agents for live data collection is expensive or unsafe (e.g., tool-use agents, robotic manipulation, autonomous driving fleets). The CTDE paradigm mirrors the centralized-training/decentralized-inference pattern common in LLM agent frameworks, where a global critic or value model supervises independently acting agents. The stationary distribution regularization technique (constraining occupancy measures rather than individual actions) offers a principled alternative to the per-action KL penalties used in RLHF and offline preference optimization, and may inform future multi-agent RLHF methods. The formal credit assignment guarantees (Proposition 4.3) are relevant to any system requiring consistent global-to-local policy decomposition in cooperative agent teams.