Skip to content

Multi-Agent Reinforcement Learning with Communication-Constrained Priors

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes a MARL framework (CC-MADDPG) that handles lossy inter-agent communication by modeling communication constraints as learnable priors and using a Dual Mutual Information Estimator (Du-MIE) to simultaneously maximize the influence of lossless messages and suppress the influence of lossy messages on agent decisions. The approach reshapes the global reward with MI-derived signals and demonstrates strong robustness across Markov-based and distance-based communication failure benchmarks.

Problem

Existing communication-based MARL methods assume lossless or bandwidth-limited-but-reliable channels, making them brittle under real-world lossy conditions (noise, delay, packet drop). Prior lossy-communication methods are scenario-specific and lack a unified formalism that (1) generalizes across different failure modes and (2) explicitly separates the behavioral impact of valid vs. corrupted messages.

Method

The framework operates in three stages:

  1. Communication-Constrained Prior Modeling: A binary link variable ι_{ij} ∈ {0,1} characterizes whether a message is lossless (1) or lossy (0). For a given deployment scenario, this prior is estimated via Markov-based dropout, distance-based attenuation, or environment-specific pretraining.

  2. Du-MIE (Dual Mutual Information Estimator): Two asymmetric MI estimators operate on replay buffer subsets partitioned by ι:

  3. JSD estimator (lower bound): maximizes I(m^{ji}; a^i) for lossless messages M⁺, encouraging agents to exploit valid information.

  4. CLUB estimator (upper bound): minimizes I(m^{ji}; a^i) for lossy messages M⁻, suppressing corrupted message influence.

  5. Reward Shaping: The global reward is augmented as r̃_t = r_t + α·Σ ι^{ji} I_JSD − β·Σ (1−ι^{ji}) I_CLUB, folding MI signals into CTDE-based policy optimization (compatible with both value-based and policy-based MARL).

Key Contributions

  • A unified Dec-POMDP extension with a binary communication link variable I = {ι_{ij}} that abstracts diverse real-world lossy-communication scenarios.
  • Du-MIE for constrained communication: a loss L_CC(θ₁,θ₂) that jointly trains JSD and CLUB estimators conditioned on message reliability labels.
  • MI-based reward shaping that integrates communication quality into cooperative policy learning without requiring algorithm-specific changes.
  • Empirical validation across four MPE tasks under both Markov-Based (MBC) and Distance-Based (DBC) communication constraints.

Results

  • Simple_Tag (Heavy DBC): CC-MADDPG scores 138.0 vs. FC-MADDPG 1.5, Dropout-MADDPG(0.2) 68.7, MADDPG 71.4.
  • Simple_Spread: CC-MADDPG achieves −127.6 to −129.4 across all constraint levels vs. FC-MADDPG degrading to −191.9 under heavy DBC.
  • Ablation (Simple_Tag, Heavy DBC): Baseline (no MI) = 68.7; Variant 1 (JSD only) = 81.7; Variant 2 (CLUB only) = 120.4; Full model = 138.0 — confirming synergy of dual-direction optimization.
  • Prior matching (Table 2): Test-environment-matched priors outperform generic dropout-0.2 priors (e.g., Simple_Spread Medium DBC: −98.0 vs. −127.6).
  • CC-MADDPG performance is largely invariant to MBC loss severity (light/medium/heavy), suggesting Markov-based packet loss in 25-step episodes may not produce sufficient disruption to differentiate levels.

Limitations

  • Experiments limited to MPE (Multi-Agent Particle Environments) with small agent counts; scalability to larger-scale or continuous-control tasks is unverified.
  • Framework is validated only on MADDPG (policy-based CTDE); extension to value-based methods (e.g., QMIX) is proposed as future work but not demonstrated.
  • The binary ι_{ij} link model is a simplification; it does not capture partial degradation (e.g., corrupted but decodable messages).
  • Fixed 25-step episode length may obscure the effect of Markov-based communication fluctuation, acknowledged in the paper.
  • Prior specification requires environment knowledge or pretraining; in fully unknown deployments, only generic dropout priors are available, leaving a performance gap.

Relevance to Agentic AI / LLM Agents

As LLM-based multi-agent systems increasingly operate over unreliable channels (network APIs, message queues, asynchronous pipelines), the challenge of degraded inter-agent communication is directly analogous to the problem addressed here. The Du-MIE reward-shaping approach offers a principled mechanism for agents to discount corrupted or delayed context without redesigning the core policy — a property valuable for robust agentic orchestration. The generalized communication-constraint prior framework also connects to the broader agenda of building agents that degrade gracefully rather than catastrophically when coordination infrastructure is imperfect.