Skip to content

Bi-Level Knowledge Transfer for Multi-Task Multi-Agent Reinforcement Learning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

BiKT proposes a two-level knowledge transfer framework for multi-task MARL that extracts both individual agent skill embeddings (via VAE) and team-level cooperative tactic embeddings (via VQ-VAE codebook) from offline trajectories. A Bi-Level Decision Transformer (BDT) sequentially selects a tactic from the codebook and then infers individual skills conditioned on it, enabling zero-shot generalization to unseen tasks without online fine-tuning.

Problem

Existing multi-task MARL methods transfer only individual agent skills across tasks, ignoring team-level coordination patterns. This is insufficient because the same underlying tactic (e.g., "focus fire") manifests as different skill combinations when agent counts vary across tasks, and networks that rely solely on individual skill transfer fail to generalize to structurally distinct unseen tasks, especially under low-quality offline data.

Method

BiKT proceeds in three stages trained sequentially on offline multi-task data:

  1. Individual skill learning: A VAE with a shared skill encoder p_skill(·|s, a, i) compresses global state and joint actions into per-agent latent skill embeddings z^i ∈ R^Ns; an action decoder q_act(·|τ^i, z^i) reconstructs actions. Trained with ELBO (reconstruction + KL divergence against uniform prior).

  2. Cooperative tactic codebook: A VQ-VAE maps the concatenated skill embeddings [z^1,...,z^N] and global state through a tactic encoder p_tac into a continuous embedding ĉ_t, which is discretized via nearest-neighbor search into a finite tactic codebook C = {c^k}^K_{k=1}. A skill decoder q_skill(·|τ^i, c^k) reconstructs individual skills from a tactic. A diversity penalty discourages codebook collapse.

  3. Bi-Level Decision Transformer (BDT): An autoregressive transformer policy that first predicts the tactic codebook index I^c_t, retrieves the corresponding tactic embedding, then generates individual skill z^i_t conditioned on it. Entity-wise decomposition of observations ensures input dimension compatibility across tasks with varying agent/entity counts.

Key Contributions

  • Bi-level knowledge transfer framework (individual skills + team tactics) for multi-task MARL from offline data.
  • Cooperative tactic codebook (VQ-VAE-based) that abstracts coordination patterns invariant to agent-count variation across tasks.
  • Bi-Level Decision Transformer policy that infers tactics before skills in an autoregressive sequence, extending DT-based MARL from single-task to multi-task zero-shot generalization.
  • Diversity regularization term preventing tactic codebook collapse and promoting coverage of diverse coordination modes.

Results

  • SMAC Marine-Hard (Expert, unseen tasks): BiKT achieves 99.3%/99.9%/99.4% on 4m/5m/10m vs. HiSSD's 99.2%/99.2%/98.4%; on harder tasks like 12m: BiKT 99.0% vs. HiSSD 75.5%, ODIS 58.6%.
  • SMAC Marine-Hard (Medium-Expert, unseen tasks): BiKT 98.1%/98.8%/96.5%/95.6% on 4m/5m/10m/12m vs. HiSSD 91.1%/98.3%/96.4%/88.4%.
  • CN Expert (unseen CN-3/CN-5): BiKT 93.8%/28.2% vs. HiSSD 89.8%/25.3% vs. ODIS 85.6%/20.0%.
  • Baselines: UPDeT, ODIS, HiSSD on SMAC and MPE; BiKT consistently superior on most unseen-task splits, with smaller gains on asymmetric tasks (e.g., 7m_vs_8m, 8m_vs_9m remain challenging for all methods, <70%).

Limitations

  • Performance on heavily asymmetric tasks (e.g., 7m_vs_8m, 8m_vs_9m, 13m_vs_15m) remains low for all methods including BiKT (e.g., 68.0%, 50.0%, 4.2% on Marine-Hard Expert), indicating generalization to large enemy-count asymmetries is unsolved.
  • Ablation study is relegated to the appendix; the main paper provides no quantitative insight into the contribution of each BiKT component.
  • Tactic codebook size K is a fixed hyperparameter; no analysis of sensitivity to this choice is provided in the main text.
  • Restricted to cooperative Dec-POMDP settings; competitive or mixed tasks are not evaluated.
  • Relies on high-quality offline datasets collected by pre-trained policies; behavior under very low-quality data (e.g., random policy) is not characterized.

Relevance to Agentic AI / LLM Agents

BiKT addresses a core challenge in multi-agent systems: reusing learned coordination knowledge across structurally different tasks without re-training, which is directly analogous to the tool-reuse and skill-transfer problems in LLM-based agent frameworks. The hierarchical abstraction—skills for individual action patterns, tactics for team coordination—mirrors the plan/subplan decomposition common in LLM agent architectures. The use of a discrete codebook for team-level strategies provides a precedent for grounding emergent agent coordination in symbolic, inspectable representations. For researchers tracking agentic AI, BiKT's zero-shot generalization via offline trajectory distillation is relevant to training-free agent adaptation and cross-task deployment of multi-agent LLM systems.