High-order Interactions Modeling for Interpretable Multi-Agent Q-Learning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

QCoFr introduces a value decomposition framework for cooperative MARL that uses continued fraction neural networks (CFN) to model arbitrary-order agent interactions with only O(n) complexity, paired with a variational information bottleneck (VIB) to extract task-relevant latent information for credit assignment. The result is a system that is both more expressive than QMIX-style mixers and intrinsically interpretable, with explicit attribution to individual agents and coalitions.

Problem

Existing value decomposition methods in MARL either model only low-order interactions (VDN: additive, first-order; QMIX: implicit, opaque monotonic mixer) or suffer combinatorial explosion when attempting explicit high-order modeling (NA²Q scales O(2ⁿ) with agent count). Post-hoc explainability tools (Shapley attribution, masked attention) cannot recover temporal or relational dynamics. There is no prior method that simultaneously achieves arbitrary-order interaction modeling, linear agent-count complexity, and intrinsic interpretability.

Method

Continued Fraction Q-Network (CFN) mixer: The joint action-value function Qtot is expressed as a weighted sum of l "ladder" modules, each a truncated continued fraction over individual agent Q-values. A depth-d truncation provides a Padé approximant that exactly captures agent interactions up to order d, with total complexity O(n). The non-negative activation 1/max(|z|, δ) at each layer enforces the IGM principle.

Variational Information Bottleneck (VIB) for credit assignment: Each agent's GRU hidden state hᵢ is encoded into a Gaussian latent mᵢ via a VIB objective that maximizes I(m, u*) while minimizing I(m, h). The resulting assistive representation m, combined with global state s, is used to compute softmax credit weights αk for each CFN ladder via learned projections wₘ and wₛ. This cuts the confounding path from s to Qtot.

Training: End-to-end TD loss + VIB loss (LVIB = cross-entropy prediction term + β·KL divergence term).

Key Contributions

• Continued fraction neural network mixer that captures d-order agent interactions at O(n) complexity, avoiding combinatorial explosion
• Formal proof that depth-d CFN truncation satisfies the Padé approximation condition: f(Q) − Rd(Q) = O(Q^(d+1))
• Universal approximation proof for linear combinations of continued fraction modules
• VIB-based assistive information module that decouples latent credit signals from global state confounding
• Intrinsic interpretability: explicit interaction coefficients βᵢⱼ... reveal individual and coalition contributions without post-hoc attribution

Results

• SMAC (6 maps, including super-hard 3s5z_vs_3s6z and 6h_vs_8z): QCoFr achieves superior win rates across almost all scenarios vs. VDN, QMIX, QPLEX, CW-QMIX, CDS, SHAQ, GoMARL, ReBorn, NA²Q (5 random seeds)
• SMACv2 (3 scenarios: zerg/protoss/terran 5v5): QCoFr "significantly better than other algorithms across all scenarios"; GoMARL performs worst due to slow grouping convergence
• LBF (lbf-3-3, lbf-4-2): Competitive performance; advantage less pronounced due to simpler task structure limiting high-order interaction utility
• Ablation (2c_vs_64zg, 3s_vs_5z): d=2 yields significant improvement over d=1; performance degrades beyond task-optimal depth (e.g., best at d=6 on 6h_vs_8z)
• Ablation (CFN structure): QCoFr-D (purely additive, no inter-agent interactions) degrades substantially on 6h_vs_8z; VIB integration improves both convergence speed and final performance
• Diversity: QCoFr yields lower cosine similarity among agent Q-values under identical observations compared to VDN and QMIX, indicating more specialized role differentiation

Limitations

• CFN depth d is fixed per task and selected by hand; suboptimal or wasteful depth requires manual tuning per environment
• Authors acknowledge future work should explore adaptive depth mechanisms that adjust per state or during training
• Performance advantage on simpler tasks (LBF) is marginal, suggesting high-order modeling adds most value only in complex coordination settings

Relevance to Agentic AI / LLM Agents

This work advances the formal understanding of how cooperative agents should structure credit assignment and interaction modeling — a core challenge in any multi-agent system where agents must coordinate without explicit communication. The interpretability angle (explicit coalition weights, interaction order attribution) is directly relevant to building auditable multi-agent pipelines where human operators need to understand why agents divide labor as they do. The VIB approach to filtering noisy observations and extracting task-relevant latent signals parallels challenges in LLM-based agent architectures where agents must distill relevant context from noisy histories. The O(n) complexity scaling is practically important for deploying cooperative MARL in real-world agentic systems with many parallel actors.