Skip to content

Conditional Diffusion Model for Multi-Agent Dynamic Task Decomposition

🕒 Published (v1): 2025-11-17 08:46 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CD³T is a two-level hierarchical MARL framework that uses a conditional diffusion model to learn latent action representations, which are then clustered into subtasks and used to drive both a high-level subtask selector and low-level subtask-specific policies. The diffusion model's high representational capacity in continuous, high-dimensional spaces enables more distinguishable subtask discovery than MLP-based alternatives. Experiments on LBF, SMAC, and SMACv2 show CD³T outperforms nine baselines, especially on hard and super-hard cooperative scenarios.

Problem

Cooperative MARL over long horizons requires agents to explore a combinatorially large joint action-observation space under partial observability. Existing task-decomposition and role-learning methods rely on simple MLP structures to extract action representations, which limits subtask diversity and requires the representational capacity to scale with environment complexity. Continuous re-clustering at every timestep (e.g., ACORM) is effective but prohibitively expensive.

Method

CD³T operates in two phases:

  1. Action representation pretraining (50K steps): A UNet with cross-attention denoising network learns latent action embeddings z_a by reconstructing noisy action encodings conditioned on the agent's local observation o_i and other agents' one-hot actions a_{-i}. Auxiliary heads predict next observations and global reward from z_a, tying representations to environmental effects (losses L_d, L_p, combined as L).

  2. Subtask decomposition and hierarchical policy learning: K-means clustering is applied once (at 50K steps) over the collected z_a to define g subtasks. The subtask representation z_φ for each subtask is the mean of its member action embeddings. A high-level subtask selector (GRU + MLP encoder) assigns each agent to a subtask every ΔT timesteps via Q^φ_i = z_τ^T z_φ. Within the selected subtask, low-level agents act in the restricted action space with shared parameters. Both levels use a multi-head dot-product attention mixing network that incorporates z_φ (high level) or z_a (low level) alongside global state s for value factorization (IGM-compliant credit assignment).

Key Contributions

  • A conditional diffusion model used as the backbone for multi-agent action representation learning, replacing simple MLPs to capture multimodal action distributions.
  • A one-time k-means subtask discovery scheme that avoids per-step clustering cost while still producing semantically meaningful subtask decompositions.
  • A subtask-aware multi-head attention mixing network that injects subtask semantics into joint Q-value factorization for better credit assignment.
  • Empirical visualization showing CD³T reduces effective per-agent action space more aggressively than RODE on hard SMAC scenarios.

Results

  • LBF (4-agent/2-food; 3-agent/3-food): CD³T achieves the highest or near-highest average test return among 9 baselines (VDN, QMIX, QTRAN, QPLEX, CDS, RODE, GoMARL, ACORM-oc, DT2GS) within 1M timesteps.
  • SMAC (8 scenarios — easy/hard/super-hard): CD³T reaches the highest win rate on nearly all maps; on super-hard maps (3s5z_vs_3s6z, corridor, 6h_vs_8z) the margin over GoMARL, CDS, and RODE is large; ACORM-oc collapses without continuous reclustering.
  • Ablation (SMAC): Removing the diffusion model (replaced by MLP) causes the largest win-rate drop, especially on hard/super-hard maps; removing subtask-based attention also degrades performance; more subtask clusters (3→5) monotonically improves results.
  • Action space reduction: On complex scenarios (5m_vs_6m, 6h_vs_8z, 3s5z_vs_3s6z), CD³T achieves tighter and lower average action dimensionality than RODE, with QMIX always using the full space.
  • All results are mean ± std over 5 random seeds.

Limitations

  • The number of subtasks g is a tunable hyperparameter requiring manual search (tested 3–5); no automatic selection mechanism.
  • Subtask decomposition is fixed after the initial 50K timesteps; the clustering cannot adapt if the environment distribution shifts significantly after pretraining.
  • The diffusion model pretraining adds offline data collection cost and an additional UNet training loop; wall-clock overhead is not fully quantified.
  • Evaluated only on cooperative MARL benchmarks (LBF, SMAC, SMACv2); generalization to competitive, mixed, or real-world multi-agent settings is unvalidated.
  • The ΔT subtask reassignment interval is another hyperparameter whose sensitivity is not analyzed.

Relevance to Agentic AI / LLM Agents

CD³T advances the machinery for hierarchical, multi-agent task decomposition — a core challenge in building agentic systems that must coordinate to solve long-horizon goals. The idea of using a generative model (diffusion) to learn a latent semantic action space, then clustering it into specialized roles, is directly analogous to how LLM-based agent frameworks assign tool-use subtasks or roles to sub-agents. The subtask-aware credit assignment mechanism (injecting semantic subtask embeddings into a joint value mixer) points toward principled ways to attribute outcomes across heterogeneous agents in multi-agent LLM pipelines. For researchers building agentic systems, CD³T provides a concrete, empirically validated template for automatic role discovery without hand-engineered task decompositions.