In-Context Fully Decentralized Cooperative Multi-Agent Reinforcement Learning¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

RAC (Return-Aware Context) addresses fully decentralized cooperative MARL—where agents have no access to other agents' actions—by formalizing each agent's local dynamics as a contextual MDP and using discretized episodic return values as context proxies for the joint policy. This simultaneously resolves non-stationarity (via context-conditioned value functions) and relative overgeneralization (via individual optimistic marginal values), without requiring any inter-agent communication.

Problem¶

Fully decentralized MARL forbids access to other agents' actions during both training and execution—a realistic constraint for privacy-sensitive or bandwidth-limited multi-robot settings. This causes two compounding pathologies: (1) non-stationarity, because each agent's local transition and reward distributions change as other agents update their policies; and (2) relative overgeneralization, where per-agent Q-values are biased downward by other agents' suboptimal exploratory actions, preventing convergence to optimal joint actions. Prior work addresses each issue in isolation; no fully decentralized method handles both simultaneously because modeling the joint policy without joint-action access is intractable.

Method¶

RAC frames each agent i's local task as a contextual MDP where contexts correspond to distinct joint policies of all other agents. Under the assumption that joint policies change only between episodes (not within), RAC uses the discretized episodic return as a proxy for the joint policy: the scalar return R(τ^i) is binned into m intervals via floor division, and the one-hot bin index c_κ serves as the context.

Two learned components per agent: 1. Context-based value function Q^i(s, c_κ, a^i): trained with standard TD loss on augmented transitions (s, a^i, c_κ, r, s'), producing stationary updates because context captures the current joint policy. 2. Individual optimistic marginal value ϕ^i(s, a^i) = max_{c_κ} Q^i(s, c_κ, a^i): the max over all contexts, implementing an optimistic belief that others cooperate, thereby eliminating relative overgeneralization.

Because episodic returns are only available at episode end, a separate standard value function Q^i_S is learned via TD loss and a KL-divergence supervision loss that distills ϕ^i into Q^i_S for within-episode action selection and return-space exploration.

Key Contributions¶

Formalizes per-agent non-stationarity in fully decentralized MARL as a contextual MDP, attributing dynamics shifts to context (joint policy) switches between episodes.
Proposes using discretized training episodic return as a fully decentralized, scalable proxy for the unobservable joint policy—no communication required.
Derives individual optimistic marginal value ϕ^i as max over context-conditioned Q-values, simultaneously addressing relative overgeneralization without requiring joint action access.
Introduces a two-network scheme (Q^i + Q^i_S with supervision KL loss) to handle sparse return-space coverage during early training.

Results¶

Matrix game: RAC converges to the optimal joint action (reward=8); IQL stalls at suboptimal (reward=6); Hysteretic-Q also reaches 8 but breaks on harder tasks; I2Q reaches 8.
Predator and prey (8v8, partial observability): RAC achieves ~+40 reward; IQL gets 0; Hysteretic-Q ~+20 with poor efficiency; I2Q fails (0).
SMAC (7 maps—2s3z, 3s5z, 3s_vs_4z, 3s_vs_5z, 5m_vs_6m, 10m_vs_11m, 2s_vs_1sc): RAC significantly outperforms all fully decentralized baselines on 5m_vs_6m, 10m_vs_11m, 3s5z, 3s_vs_4z; matches IQL on easier maps.
RAC outperforms centralized-training QMIX and VDN on matrix game and predator-prey; underperforms them on harder SMAC maps due to partial-observability and coverage limitations.
Baselines compared: IQL, Hysteretic Q-learning, I2Q (fully decentralized); QMIX, VDN (centralized training, for reference).

Limitations¶

Requires full coverage of the episodic return space; complex tasks make early coverage infeasible, reducing to the two-network workaround.
Manual discretization of return bounds (R_min, R_max, m) requires prior knowledge and introduces variance; no adaptive scheme.
Only handles the case where context changes between episodes (Case 2); cannot address intra-episode non-stationarity (Case 1, policies updating at every step).
Optimistic marginal value computation scales with m; large m requires sampling-based approximation (e.g., CEM), adding overhead.
Performance lags behind centralized-training methods (QMIX/VDN) on complex SMAC maps, partly due to partial observability handling.

Relevance to Agentic AI / LLM Agents¶

While RAC operates in classical MARL rather than LLM settings, it directly addresses a structural challenge relevant to any multi-agent system where agents cannot observe each other's actions or reasoning processes—a realistic constraint for independently deployed LLM agents. The core insight—using a scalar signal (episodic return) as a proxy for unobservable joint behavior to enable decentralized coordination—parallels how LLM agents might infer collaborator intent from shared outcome signals rather than direct action visibility. The contextual MDP framing also provides a principled vocabulary for diagnosing non-stationarity in multi-agent LLM pipelines where each agent's effective environment shifts as others are updated or retrained.