HCPO: Hierarchical Conductor-Based Policy Optimization in Multi-Agent Reinforcement Learning

🕒 Published (v1): 2025-11-15 09:19 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HCPO introduces a hierarchical conductor-based framework for cooperative MARL where a centralized conductor broadcasts discrete latent instructions that condition per-agent policies, expressing the joint policy as a Gaussian-mixture-like expectation: π_mar(a|s) ≜ E_{M~w(·|s)}[π(a|s,M)]. A two-level trust-region update—first optimizing the conductor, then sequentially updating agents conditioned on each instruction—yields a provable monotonic joint policy improvement guarantee. Local conductor distillation via cross-entropy enables fully decentralized execution without inter-agent communication.

Problem

Existing CTDE MARL methods (HATRPO, A2PO, MAPPO) factorize the joint policy as a product of individual policies (π_joint = ∏ π^i), which fundamentally caps the expressive capacity of the joint policy and forces exploration through uncoordinated independent agents. This product-of-marginals assumption makes it difficult to discover high-value coordinated joint behaviors, particularly in large action spaces.

Method

The joint policy is redefined as a mixture: π_mar(a|s) = E_{M~w(·|s)}[π(a|s,M)], where M is a discrete instruction (K choices) sampled from a learned centralized conductor w(·|s), and each agent's policy π^i(a^i|s,M) is conditioned on the broadcast instruction.

The joint advantage decomposes as A_π_mar(s,a) = A_π_mar(M|s) + A_π_mar(a|s,M) (Eq. 7), enabling a two-level update: 1. Conductor update: maximize the instruction advantage E[A(M|s)] subject to a KL trust-region constraint on w. 2. Agent update: for each instruction M^j, sequentially update each agent π^{i_l} using the conditional advantage decomposition (Lemma 2), weighted by w(M^j|s), subject to per-agent KL constraints.

Theorem 1 provides a lower bound on J(π̄_mar) - J(π_mar) decomposed into conductor-KL and per-instruction agent-KL penalties, guaranteeing monotonic improvement. For decentralized execution, the centralized conductor is distilled into per-agent local conductors w_i(·|o_i) via cross-entropy loss, eliminating communication at test time.

Key Contributions

• Conductor-based joint policy expression: π_mar(a|s) = E_{M~w(·|s)}[π(a|s,M)], strictly more expressive than the product-of-marginals assumption used by prior work.
• HCPO algorithm: two-level sequential policy update (conductor first, then agents per instruction) with dual trust-region constraints.
• Monotonic improvement guarantee: proven without relying on QMIX's monotonicity hypothesis on the action-value function.
• Communication-free execution: centralized conductor distilled to local conductors via cross-entropy, preserving CTDE benefits.
• Empirical validation: evaluated on SMAC (5 maps), MA-MuJoCo, and MPE against HATRPO, HAPPO, A2PO, HAA2C, and MAVEN.

Results

• SMAC: HCPO is the first algorithm to achieve ≥90% win rate across all 5 test maps; achieves lowest standard deviation across runs among all baselines.
• MA-MuJoCo HalfCheetah-v2-2×3: ~23.42% higher final returns than next-best baseline (HAA2C).
• MA-MuJoCo Walker2d-v2-6×1: superior early-training exploration demonstrated by higher state-space entropy and larger average nearest-neighbor distance in t-SNE projections vs. HATRPO and A2PO.
• MPE: faster policy improvement in the 0–2M step window; higher stability and robustness than HATRPO and A2PO.
• Ablation: local conductors achieve median return comparable to a fully centralized conductor, substantially outperforming the no-conductor variant; random (non-learning) conductor performs worse than learned, confirming the learned instruction distribution is essential.

Limitations

• On-policy algorithm only; the hierarchical conductor framework has not been extended to off-policy settings, limiting sample efficiency.
• Number of instructions K is a sensitive hyperparameter requiring tuning to balance performance and computational overhead.
• Local conductor distillation is an approximation; any mismatch between centralized and local conductors is unquantified during execution.
• Evaluated only on fully cooperative tasks with shared reward; behavior in mixed cooperative-competitive settings is untested.

Relevance to Agentic AI / LLM Agents

HCPO's conductor-agent hierarchy directly mirrors the orchestrator-subagent pattern central to multi-agent LLM systems: a high-level planner (conductor) issues discrete strategic instructions that condition the behavior of lower-level executors (agents), a structure increasingly common in tool-use and agentic pipelines. The formal treatment of expressive joint policy factorizations is relevant to researchers designing multi-agent LLM systems where subagent coordination cannot be reduced to independent sampling. The monotonic improvement guarantee and trust-region framing offer a theoretical scaffold for analyzing policy-gradient-based fine-tuning of hierarchical agent systems. Additionally, the cross-entropy distillation approach for enabling communication-free execution parallels challenges in deploying coordinated LLM agents without a persistent shared state.