MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning

🕒 Published (v1): 2025-02-25 18:33 UTC · Source: Arxiv · Venue: ACL · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MAPoRL is a post-training paradigm that co-trains multiple LLMs jointly via multi-agent reinforcement learning (MARL) to elicit genuine collaborative behavior in multi-agent debate frameworks. The core insight, supported by a game-theoretic analysis, is that single-agent RL training or SFT is structurally insufficient for inducing collaboration because a non-strategic opponent cannot promote cooperative equilibria. Co-training with an influence-aware reward signal and explicit collaboration incentives yields agents that improve across multi-turn interactions, whereas off-the-shelf LLMs stagnate.

Problem

Multi-agent LLM collaboration frameworks (e.g., debate) rely on prompting pre-trained models, which were never optimized for cooperative interaction. Empirically, multi-agent debate with off-the-shelf LLMs does not consistently benefit from additional turns (Huang et al., 2024). SFT on multi-agent interaction traces is also inadequate: mimicking logged interactions does not produce strategic cooperation. The theoretical gap is that collaboration is an emergent equilibrium property of jointly-optimized policies, not a fixed behavior transferable from data.

Method

Framework: A collaborative debate system of \(A\) agents over \(T\) turns. Agent \(i\)'s turn-\((t+1)\) response is: $\(s^{(t+1)i} = \text{LLM}_i\!\left(q \oplus_{j \in [A], t' \in [t]} s^{t'j}\right)\)$

Reward — Influence-aware Verification Reward: $\(R_\theta(q, s^a_t) = \mathbb{E}\!\left[\frac{\text{Verifier}(q,s^a_t)}{\sum_{t' \in [t,T]} \gamma^{t'-t}} + \sum_{t' \in [t+1,T]} \sum_{j \in [A]} \frac{\gamma^{t'-t}}{A} \text{Verifier}(q,s^{t'j})\right]\)$ The reward credits the agent for both its current answer quality and the downstream influence it exerts on all future responses by all agents.

Optimization: Multi-agent PPO (MAPPO; Yu et al., 2022) with decentralized policies and per-turn value functions. Token-level advantage is estimated via GAE; a KL penalty against a frozen reference model (\(\lambda_\text{KL}\)) prevents policy collapse.

Reward shaping: Four scalar incentive parameters (\(\alpha_0, \alpha_1, \beta_0, \beta_1\)) reward/penalize an agent's own answer transitions (correct↔incorrect relative to majority) and its influence on shifting other agents' majority opinion, operationalizing critical reasoning and persuasion skills.

Training infrastructure: Phi-3-mini-128k-instruct (3.4B), Qwen2.5-3B, Llama-3-8B fine-tuned with QLoRA under quantization. Verifier trained separately on held-out splits to prevent overfitting.

Key Contributions

• Game-theoretic proof (via coordination-game reduction and entropy-regularized Nash equilibrium analysis) that single-agent training cannot produce cooperative equilibria when the opponent is non-strategic.
• MAPoRL: the first RL-based multi-LLM co-training paradigm applied to collaborative debate, as opposed to contemporaneous iterative-SFT approaches.
• Influence-aware verification reward that propagates quality signals across all agents and future turns within a single scalar.
• Four-parameter reward-shaping scheme (α, β) encoding corrective reasoning and persuasion incentives.
• Empirical demonstration that MAPoRL performance scales with turns while off-the-shelf collaboration does not, with cross-domain generalization (train on ANLI, evaluate on GSM8K).

Results

• Off-the-shelf Phi-3 (GSM8K): accuracy flat across turns (T2→T3 net improvement: +0.47%); on ANLI net improvement: −1.72% — confirming the collaboration-stagnation baseline.
• MAPoRL-trained (GSM8K): net improvement T2→T3: +4.01%; on ANLI: +1.09% — agents improve with more turns.
• Single-question accuracy (no collaboration context, same model weights): Phi-3 0.609/0.451, MAPoRL-T2 0.604/0.458, MAPoRL-T3 0.611/0.453 (GSM8K/ANLI) — confirming gains are collaboration-derived, not domain-knowledge memorization.
• Transition analysis (GSM8K, Figure 3): MAPoRL reduces Wrong→Wrong persistence and increases Wrong→Right recovery compared to off-the-shelf baselines; net improved metric 12.03% for a specific incentive configuration.
• Toy MARL experiments (T=10,20) show jointly-optimized agents converge to high collaboration rates; single-agent best-response to a fixed π(q)=0.6 opponent still avoids collaboration.

Limitations

• Experiments use quantized small models (3.4B primary) with QLoRA due to compute budget; scalability to frontier-scale models is untested.
• The collaborative debate framework is the only multi-agent structure evaluated; generality to other harness topologies (hierarchical, tool-using, critic-actor) is assumed but not demonstrated.
• Verifier training data and MAPoRL training data must be disjoint to avoid overfitting; this constrains data efficiency.
• Paper text is truncated before Experiments 2–4 results are fully reported, so incentive-parameter ablations and heterogeneous-agent results are only partially available in the excerpt.
• Final answer selection relies on majority voting, which can be noisy under small \(A\).

Relevance to Harnesses / Meta-Harnesses

MAPoRL treats the entire multi-agent pipeline (the debate harness) as the differentiable training environment rather than a static orchestration scaffold, making it one of the clearest examples of a harness becoming a first-class object of optimization. The influence-aware reward is effectively a meta-level signal that evaluates the quality of the harness's collective output trajectory, not just individual agent steps — a design principle directly applicable to meta-harnesses that coordinate heterogeneous agents. The result that harness topology (number of agents, turns, interaction graph) must be co-optimized with agent policies is a strong argument for treating harness structure as a hyperparameter in any meta-harness design. For researchers building harnesses that orchestrate LLM sub-agents, MAPoRL provides both a theoretical grounding (coordination-game equilibria) and a practical recipe (MAPPO + influence-aware reward) for moving from static prompt-based orchestration to learned, self-improving multi-agent loops.