Skip to content

Adaptive Collaboration with Humans: Metacognitive Policy Optimization for Multi-Agent LLMs with Continual Learning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Metacognitive policy optimization for continual human-AI collaborative multi-agent learning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HILA (Human-In-the-Loop Multi-Agent Collaboration) breaks the "closed-world" ceiling of autonomous multi-agent LLM systems by training agents with a metacognitive policy that decides when to solve problems autonomously and when to defer to a human expert. A Dual-Loop Policy Optimization (DLPO) algorithm combines GRPO-based RL for deferral decisions with continual supervised fine-tuning from expert demonstrations. On competition math (AMC, AIME) and code benchmarks, HILA consistently outperforms all tested autonomous MAS baselines.

Problem

Autonomous multi-agent systems (MAS) are fundamentally bounded by the knowledge encoded at pre-training time. No matter how sophisticated their internal interaction protocols (debate, topology control, workflow optimization), they cannot acquire genuinely new capabilities—they only recombine existing knowledge. Prior human-in-the-loop work treats humans as passive oracles, relies on heuristic triggers (e.g., low-confidence thresholds) rather than learned deferral policies, and applies feedback as one-time fixes rather than signals for lasting capability growth.

Method

HILA formalizes human–agent collaboration as a Metacognitive Markov Decision Process (Meta-MDP) over high-level cognitive actions rather than token generation.

Cognitive state \(s_t\) encodes three information sources plus optional structured signals: $\(s_t = \text{concat}\!\left(x_t^\text{task},\, x_t^\text{self},\, x_t^\text{peer},\, z_t^\text{soc},\, z_t^\text{mon},\, z_t^\text{ctrl}\right)\)$ where \(z^\text{soc}\), \(z^\text{mon}\), \(z^\text{ctrl}\) are lightweight rule-based cues for social consensus, metacognitive monitoring, and cognitive control.

Action space \(\mathcal{A} = \{a_\text{eval}, a_\text{create}, a_\text{defer}\}\): - EVAL: endorse a peer's existing solution (exploitation) - CREATE: generate a novel solution (exploration) - DEFER: invoke the human expert; adopt \(y_\text{human}\) as output

Dual-Loop Policy Optimization (DLPO): - Inner loop (RL): GRPO over \(\pi_\theta(a|s_t)\) with a cost-aware reward: $\(R(s_t, a_t) = \begin{cases} R_g(\hat{y}) & a_t = \text{EVAL} \\ R_g(\hat{y}) - C_\text{create} & a_t = \text{CREATE} \\ R_g(y_\text{human}) - C_\text{defer} & a_t = \text{DEFER} \end{cases}\)$ with \(C_\text{defer} > C_\text{create} \geq 0\). Advantages are computed by group-centering rewards; the objective adds a KL penalty and entropy bonus. - Outer loop (SFT): when DEFER fires, the human demonstration \(y_\text{human}\) becomes an SFT sample minimizing cross-entropy, expanding the model's knowledge boundary. - Joint objective: \(\mathcal{L}_\text{total} = \mathbb{E}\!\left[\mathcal{L}_\text{Inner} + \lambda_\text{sft} \cdot \mathbf{1}(a_t = \text{defer}) \cdot \mathcal{L}_\text{SFT}\right]\)

GPT-4o-mini is used as the proxy human expert in experiments.

Key Contributions

  • HILA framework: principled Human-in-the-Loop MAS with a learned metacognitive policy (\(\pi_\theta\) over EVAL/CREATE/DEFER) rather than heuristic thresholds.
  • Dual-Loop Policy Optimization (DLPO): decouples short-term deferral optimization (inner RL/GRPO) from long-term capability growth (outer continual SFT).
  • Meta-MDP formalism: provides a theoretical grounding for human–agent collaboration as a decision process over cognitive strategies.
  • Cost-aware reward: balances task correctness against intervention costs, enabling the policy to prefer cheaper actions when equally effective.
  • Empirical demonstration that gains compound from both learned deferral and continual learning (ablation in Table 3).

Results

All results use LLaMA3-8B backbone unless noted; best autonomous MA baseline underlined in original:

  • GSM8K: HILA 89.86% vs. best baseline GPTSwarm 84.89% (+5.0 pp)
  • AMC: HILA 35.83% vs. best baseline G-Debate 20.48% (+15.35 pp)
  • AIME: HILA 9.37% vs. best baseline GPTSwarm 5.78% (+3.59 pp)
  • HumanEval: HILA 73.62% vs. best baseline G-Debate 69.89% (+3.73 pp)
  • MMLU: HILA 62.20% vs. best baseline AFlow 62.20% (tied or marginal)
  • Cross-backbone (GSM8K): HILA gains over best autonomous baseline: Qwen2.5-7B +1.89 pp, Qwen2.5-3B +3.32 pp, LLaMA3-8B +5.0 pp, LLaMA3-3B +7.53 pp — gains are largest on weaker models.
  • Ablation: HILA+DLPO (35.83 AMC) outperforms HILA+GRPO-only (32.50) and HILA+Init Policy (33.33), confirming that continual SFT contributes beyond pure RL.
  • Applying DLPO to baselines (DyLAN, Vanilla) also improves them, but HILA+DLPO remains best.

Limitations

  • The "human expert" in experiments is GPT-4o-mini, not a real human; the gap between a proxy LLM and actual expert knowledge is unstated.
  • Deferral cost hyperparameters (\(C_\text{create}\), \(C_\text{defer}\), \(\lambda_\text{sft}\)) require tuning; sensitivity analysis is deferred to appendices.
  • Continual SFT from DEFER events raises potential catastrophic forgetting concerns, which are not explicitly analyzed.
  • Real human collaboration introduces latency and availability constraints not modeled here.
  • The paper is truncated before the full Table 3 and some cross-backbone results; MMLU gains appear modest.

Relevance to Agentic AI / LLM Agents

HILA directly addresses a core architectural challenge for deployed agentic systems: when should an agent escalate to a human versus proceed autonomously? By formalizing this as a learned metacognitive policy via RL rather than a hand-coded heuristic, it moves toward agents that can self-regulate their own competence boundaries. The DLPO continual-learning loop is particularly significant for long-running agents that must improve over time from human corrections—a practical requirement for real-world deployment. This work connects to a growing line bridging RLHF-style feedback, multi-agent coordination, and human-in-the-loop systems, and offers a concrete training recipe (GRPO + SFT gated on DEFER) that could generalize beyond the mathematical reasoning benchmarks tested here.