Skip to content

Oryx: a Scalable Sequence Model for Many-Agent Coordination in Offline MARL

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Oryx is an offline cooperative MARL algorithm that combines the retention-based sequence model Sable with a sequential autoregressive formulation of implicit constraint Q-learning (ICQ) to address extrapolation error and miscoordination in many-agent settings. By decomposing joint policy updates autoregressively and conditioning each agent's update on prior agents' committed actions, Oryx achieves state-of-the-art performance on over 80% of 65 tested offline MARL datasets while scaling to 50-agent coordination tasks where prior methods collapse.

Problem

Offline MARL faces two compounding challenges: (1) extrapolation error—agents selecting out-of-distribution actions whose joint-action-space grows exponentially with agent count—and (2) miscoordination—agents trained on static historical data from suboptimal policies may develop incompatible behaviors, especially over long horizons. Prior offline MARL methods address one or the other but are typically evaluated only on few-agent, short-horizon settings and lack scalable sequential coordination mechanisms.

Method

Oryx adapts Sable's retention-based encoder–decoder architecture (replacing softmax attention with a decaying retention matrix, supporting recurrent inference and chunkwise training) with a dual-output decoder that simultaneously produces policy logits and per-agent Q-value estimates. The core contribution is an autoregressive ICQ loss derived from the multi-agent advantage decomposition theorem (Kuba et al., 2021): at each policy iteration step, agents are updated sequentially in a random permutation order, with each agent's update conditioned on the already-committed actions of preceding agents. The ICQ Bellman operator avoids explicit behavior policy estimation by computing target Q-values sampled from the dataset with implicit importance weights (1/Z(τ')) exp(Q(τ',a')/α). A counterfactual advantage baseline (COMA-style) replaces the standard V-value baseline, removing the (n-1) variance scaling term and yielding tighter gradient estimates as agent count grows. Critic and policy losses are optimized jointly end-to-end without a separate encoder value head.

Key Contributions

  • Formal derivation of sequential autoregressive ICQ policy optimization under the multi-agent advantage decomposition theorem (Theorem 1).
  • Dual-decoder Sable variant enabling simultaneous Q-value and policy estimation for counterfactual advantage computation.
  • New offline datasets for the Connector environment with up to 50 agents, released publicly via OG-MARL.
  • Empirical demonstration that all three components—autoregressive action selection, retention-based memory, and ICQ regularization—are individually necessary (T-Maze ablation).
  • State-of-the-art results on 34/43 SMAC datasets (79%), 14/16 MAMuJoCo datasets, and all 6 RWARE datasets.

Results

  • T-Maze (ablation): Oryx scores 0.99 ± 0.01 (replay) / 0.94 ± 0.03 (expert); all baselines (I-ICQ, MAICQ) and single-component ablations score 0.0.
  • Connector scaling (up to 50 agents): At 23+ agents, Oryx achieves near-expert performance vs. ~25% expert performance for MAICQ; MAICQ degrades sharply while Oryx remains robust through 50 agents.
  • SMAC: Oryx matches or surpasses SOTA on 34/43 datasets (79%) against CFCQL, MADTKD, ComaDICE, MAICQ, and others.
  • MAMuJoCo: SOTA on 14/16 datasets.
  • RWARE: SOTA on all 6 datasets, improving prior best by ~20% on several.
  • Architecture ablation (vs. MAT+ICQ): On SMAC, Oryx IQM 0.87 [0.86, 0.88] vs. MAT+ICQ 0.67 [0.66, 0.69]; optimality gap 0.23 vs. 0.38. On RWARE, Oryx IQM 0.89 vs. MAT+ICQ 0.85.

Limitations

  • Evaluated only on research benchmarks (SMAC, MAMuJoCo, RWARE, Connector); no real-world industrial validation.
  • Offline-only; no offline-to-online transfer experiments despite authors flagging this as a key use case.
  • Connector datasets generated from Sable's own replay data, potentially introducing distribution alignment that advantages Oryx.
  • Random agent permutation at each update step may reduce reproducibility and adds sensitivity to permutation variance in very large agent populations.
  • No analysis of compute/wall-clock scaling relative to agent count despite the scalability claim.

Relevance to Agentic AI / LLM Agents

Oryx demonstrates that autoregressive sequential action generation—the same structural inductive bias used by LLM-based agents—is a principled mechanism for multi-agent coordination, not just a convenient parameterization. The retention-based sequence model backbone connects directly to efficient long-context reasoning architectures relevant to agentic frameworks that must maintain coherent state over extended task horizons. For researchers building multi-agent LLM systems that operate from logged interactions (e.g., tool-use traces, dialogue histories) rather than live environment feedback, Oryx's offline autoregressive ICQ framework offers a rigorous theoretical grounding for policy extraction from static data without online exploration. The scalability results (50 agents, long-horizon sparse reward) are directly relevant to multi-agent coordination problems in LLM pipelines where many specialized agents must act sequentially without runtime interaction.