MisoDICE: Multi-Agent Imitation from Mixed-Quality Demonstrations

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MisoDICE addresses offline imitation learning in cooperative multi-agent settings where demonstrations are unlabeled and of mixed quality. The two-stage framework first uses LLMs plus preference-based RL to label trajectories by quality, then applies a novel DICE-based multi-agent IL algorithm with value decomposition to learn robust policies without reward supervision. It achieves state-of-the-art performance on SMACv1 and SMACv2 benchmarks, especially when expert data is scarce.

Problem

Prior multi-agent imitation learning algorithms assume access to large, clean expert datasets, which is impractical in realistic cooperative settings where the joint state-action space grows combinatorially with agent count. No prior work had tackled the setting of offline multi-agent IL from unlabeled, heterogeneous (mixed expert + suboptimal) demonstrations without reward annotations.

Method

Stage 1 — Multi-step trajectory labeling: 1. Sample trajectory pairs from the unlabeled dataset and query an LLM (GPT-4o) for pairwise preference labels based on semantic task context. 2. Train O-MAPL (offline preference-based MARL) on these noisy LLM preferences to learn a soft Q-function. 3. Recover a reward signal via R ≈ Q − γV, compute per-trajectory returns, and rank trajectories; the top-K become the expert dataset D_E, the rest form D_Mix.

Stage 2 — MisoDICE algorithm: - Formulates multi-agent IL as a DICE-style constrained occupancy-matching problem with a dual KL objective: minimize divergence to expert D_E while penalizing distance from the union dataset D_U. - Handles the intractable joint action space via CTDE: local value networks νᵢ(sᵢ) are combined through a linear mixing network Mϕ (proven to preserve convexity of the objective; nonlinear mixing destroys convexity per Proposition 5.2). - Occupancy ratio ρᴱ/ρᵁ is estimated via local discriminator networks aggregated by a second linear mixing network. - Policy extraction: each agent's local policy is recovered via a globally-weighted behavior cloning step; Proposition 5.4 proves local optimization is globally consistent; Proposition 5.5 gives a closed-form softmax expression for the optimal local policy as a function of local Q and V.

Key Contributions

• First offline multi-agent IL method for unlabeled mixed-quality demonstrations.
• LLM + O-MAPL two-step progressive labeling pipeline that refines coarse LLM preferences into grounded trajectory rankings.
• MisoDICE algorithm extending single-agent DemoDICE to multi-agent CTDE with linear value mixing; theoretical proofs of convexity, local-global policy consistency, and closed-form local policy expression.
• Ablation demonstrating that linear (not nonlinear) mixing is essential for stable training in offline low-data regimes.

Results

• Evaluated on SMACv2 (Protoss/Terran/Zerg, 5v5 to 20v23 scenarios) and SMACv1 (2c_vs_64zg, 5m_vs_6m, 6h_vs_8z, corridor).
• MisoDICE achieves the highest mean return in nearly all SMACv2 scenarios versus BC (β=0.0/0.5/1.0), O-MAPL, INDD (independent DemoDICE), MARL-SL, and VDN ablation.
• Example SMACv2 gains (corridor): MisoDICE 5.8±0.8 vs. best baseline VDN 4.7±0.6 vs. BC(β=1.0) 4.3±0.7.
• Ablation on top-K selection: performance peaks near k=200 expert trajectories; using k=50 or k≥800 degrades median return and increases variance.
• INDD (no coordination) consistently underperforms MisoDICE, confirming that value decomposition is necessary.
• LLM quality matters: GPT-4o vs. GPT-4o-mini and rule-based oracle alternatives evaluated in appendix.

Limitations

• Restricted to cooperative settings; extension to competitive multi-agent environments is left to future work.
• LLM-based preference generation fails in domains like MAMuJoCo where abstract physics/kinematics semantics are opaque to language models; requires a rule-based oracle fallback.
• Optimal k (number of labeled expert trajectories) is task-dependent and must be treated as a hyperparameter.
• Linear mixing network constrains the expressiveness of value factorization compared to QMIX-style nonlinear mixing.

Relevance to Agentic AI / LLM Agents

MisoDICE exemplifies a hybrid paradigm where LLMs serve as a weak semantic supervisor — generating noisy preference labels that bootstrap a downstream learned system — rather than acting as direct policy executors. This is directly relevant to researchers building agentic systems that must learn cooperative behaviors from imperfect, real-world interaction logs where quality labels are unavailable. The two-stage label-then-imitate architecture mirrors how LLM-guided reward shaping is being explored in single-agent settings, and extending it to multi-agent coordination is a meaningful step toward deploying LLM-assisted learning in team-based autonomous agents. The theoretical grounding (convexity guarantees, global-local consistency) is particularly valuable for trust-critical agentic deployments requiring predictable joint behavior.