ACC-Collab: An Actor-Critic Approach to Multi-Agent LLM Collaboration

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ACC-Collab jointly trains a two-agent team (actor + critic) to collaborate via an actor-critic learning framework, treating collaboration as a learned behavior rather than an emergent one. It introduces "Guided Collaborative Trajectories" — off-policy data generation that steers deliberation toward and away from correct answers — paired with DPO to optimize both agents. The resulting teams consistently outperform inference-only and training-based multi-agent baselines across five benchmarks.

Problem

Existing multi-agent deliberation methods (MAD, Society of Minds, DebateGPT, etc.) use off-the-shelf LLMs not trained to collaborate, relying on collaboration as an emergent property. This leads to unstable, limited improvement across deliberation rounds, especially on weaker base models. No prior work jointly trains the entire actor-critic team for collaborative problem-solving.

Method

ACC-Collab frames two-agent deliberation as a cooperative Stackelberg game and solves it via iterative best-response: alternately freeze the critic and optimize the actor, then vice versa, using DPO.

Guided Collaborative Trajectory generation (Algorithm 1): For each task \((x, y)\) and each round \(t\), three trajectories are generated in parallel: a natural trajectory \(z^{(t)}\), a guided-toward-correct trajectory \(z^{(t)}_+\), and a guided-away trajectory \(z^{(t)}_-\) (achieved by injecting the correct/wrong answer into the prompt). Reward \(r(z^{(t)}, x, y)\) is estimated via one-step Monte Carlo rollout — the mean accuracy if deliberation continues from \(z^{(t)}\). Preference pairs are selected when the improvement exceeds threshold \(\varepsilon\): $\(\Delta_y = r(z^{(t)}_{y,a}, x, y) - r(z^{(t)}_a, x, y) \geq \varepsilon\)$ These pairs train both agents with the multi-round DPO loss: $\(\mathcal{L}_{\text{DPO}} = \sum_{t=0}^{T} \mathbb{E} \left[ \log \sigma\!\left( \frac{\pi_\theta(z^{(t)}_+|\cdot)}{\pi_{\text{ref}}(z^{(t)}_+|\cdot)} - \frac{\pi_\theta(z^{(t)}_-|\cdot)}{\pi_{\text{ref}}(z^{(t)}_-|\cdot)} \right) \right]\)$

A single training round (ACC-Collab) or two rounds (ACC-Collab+) are evaluated.

Key Contributions

• First framework to jointly train an actor-critic LLM team for collaborative problem-solving.
• "Guided Collaborative Trajectories": off-policy data generation scheme that steers deliberation bidirectionally to produce high-quality preference pairs efficiently.
• Partial Trajectory Reward: Monte Carlo estimation of \(r(z^{(t)}, x, y)\) — expected final-round accuracy conditioned on round-\(t\) response — enabling credit assignment across multi-turn deliberation.
• Demonstrated superior final-round accuracy and higher per-round improvement rate vs. SoM, Persona, SFT, DebateTune, and DebateGPT across three base models (Llama-3-8B, Mistral-7B, Gemma-2-2B) and five benchmarks.

Results

• Llama-3-8B: ACC-Collab reaches 0.887 on BoolQ (vs. 0.815 DebateGPT, 0.812 SoM-2x); ACC-Collab+ reaches 0.683 on MMLU (vs. 0.654 DebateGPT); 0.952 on SCIQ vs. 0.932 DebateGPT.
• Mistral-7B: ACC-Collab+ achieves 0.893 on BoolQ (vs. 0.848 DebateGPT, 0.831 Persona), 0.672 on MMLU (vs. 0.594 SFT), 0.601 on BBH (vs. 0.480 DebateGPT).
• Gemma-2-2B: ACC-Collab reaches 0.840 on BoolQ vs. 0.812 DebateGPT; gains are more modest on MMLU (0.510 vs. 0.582 DebateGPT), indicating base-model sensitivity.
• ACC-Collab+ achieves the highest average percent improvement from round \(t=0\) to \(t=4\) across all three base models (Figure 2); baseline methods (SoM, SFT, DebateGPT) show near-zero improvement on some model/dataset pairs.
• Actor zero-shot accuracy at \(t=0\) (before any deliberation) is also higher for ACC-Collab than SFT/DebateGPT on BoolQ/Llama-3, suggesting training generalizes beyond collaboration.

Limitations

• One-step Monte Carlo rollout for reward estimation is a heuristic; learning-based reward functions are left to future work.
• ACC-Collab+ (two training rounds) can degrade performance relative to ACC-Collab on some benchmarks, requiring a hold-out set to decide when to stop.
• Evaluation is limited to classification/MCQ benchmarks (BoolQ, MMLU, BBH, SCIQ, ARC); generalization to open-ended generation is unverified.
• Only two-agent (actor + critic) teams studied; scaling to larger agent teams is not addressed.
• Prompt-guided steering used for off-policy generation relies on the base model's instruction-following capability, which may limit applicability to weaker models.

Relevance to Harnesses / Meta-Harnesses

ACC-Collab is directly relevant as a training harness for multi-agent pipelines: it provides a structured data-generation loop (guided rollouts → reward estimation → preference filtering → DPO) that orchestrates multiple model instances across alternating training phases — a meta-level control layer over agent collaboration. The Guided Collaborative Trajectory algorithm is itself a harness component: it programmatically controls agent behavior (by prompt injection), evaluates trajectory quality via Monte Carlo simulation, and selects training data — all without human labeling. For researchers building meta-harnesses, ACC-Collab demonstrates how a harness can train its own subagents to be better harness participants, closing the loop between orchestration design and agent specialization.