Latent Agents: A Post-Training Procedure for Internalized Multi-Agent Debate

🕒 Published (v1): 2026-04-27 18:06 UTC · Source: Arxiv · Venue: ACL 2026 · link

Why this paper was selected

Internalizes multi-agent debate via post-training; eliminates inference-time overhead

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

IMAD (Internalized Multi-Agent Debate) distills multi-agent debate into a single LLM via a two-stage SFT + RL pipeline, achieving competitive or superior reasoning accuracy while consuming only 6–21% of the tokens required by explicit multi-agent debate. Beyond efficiency, the paper demonstrates that internalization creates linearly separable agent-specific subspaces in activation space, enabling targeted steering to amplify or suppress individual agent behaviors.

Problem

Multi-agent debate improves LLM reasoning and reduces hallucination, but incurs prohibitive inference cost: running \(n\) model instances across \(m\) rounds produces verbose transcripts before yielding an answer. No prior work had distilled the full multi-agent debate process (intermediate rounds, not just final consensus) into a single model.

Method

Pipeline overview: Three stages — debate data collection, supervised fine-tuning (SFT), and RL-based internalization.

1. Dataset: 944 structured debate traces generated by GPT-3.5-turbo with \(n=3\) agents, \(m=2\) rounds on six-operand arithmetic. Structure tags (<|Agent 1|>, <|Round 1|>, <|endofdebate|>) are inserted. Majority-consensus filtering is applied.

2. Stage 1 — Debate Structure Learning (SFT): The base LLM is fine-tuned via autoregressive next-token prediction over the entire debate trace (not just the final answer), teaching it to autonomously generate a complete structured debate.

3. Stage 2 — Debate Internalization (RL with GRPO): The SFT model is optimized with a composite reward: $\(r(x, y) = w_{\text{fmt}} R_{\text{fmt}} + w_{\text{clip}} R(y; l)\)$

4. \(R_{\text{fmt}}\): formatting reward (presence of structure tags), with weight \(w_{\text{fmt}}\) decaying toward 0 over training to remove the incentive to verbalize debate.
5. \(R(y; l)\): correctness-with-length-clipping reward — reward 1 iff the correct answer \(y^*\) appears within the first \(l\) tokens; the clip limit is annealed from \(l_0=2000\) to \(l^*=500\) tokens. This forces the model to perform multi-perspective reasoning implicitly (in latent space) rather than textually.

Mechanistic analysis: Agent-specific steering vectors are extracted via Contrastive Activation Addition (difference-in-means on per-agent contrastive activations) and applied as \(h_\ell \leftarrow h_\ell + \alpha \cdot v_i\) to probe and control agent subspaces.

Key Contributions

• IMAD two-stage SFT+RL pipeline that internalizes multi-agent debate into a single LLM with up to 93% token reduction and matching or exceeding debate accuracy.
• Mechanistic evidence that internalization creates linearly separable agent subspaces in activation space, confirmed by steering-vector ROUGE-L comparison between IMAD and base models.
• Application: malicious agent traits (evil intent, hallucination) instilled via IMAD can be more selectively suppressed via negative steering with less collateral degradation to task performance than steering a base model.

Results

• Token efficiency: IMAD uses 6.3–21.1% of Debate's token budget across all models/tasks (5–16× reduction).
• GSM8K (LLaMA-3.1 8B): IMAD 85.20% vs. Debate 83.03% vs. Single 79.93% vs. DebateGPT 74.42%.
• BBH (LLaMA-3.1 8B): IMAD 58.53% vs. Debate 51.06% vs. Single 56.37%.
• GSM8K (Mistral Nemo 12B): IMAD 80.00% vs. Debate 61.03% (+18.97 pp).
• GSM8K (Qwen 2.5 7B): IMAD 89.67% vs. Debate 91.37% (slight underperformance).
• Agent subspace steering (ROUGE-L AUC): IMAD 0.903 vs. base 0.802 (+12.7%); average improvement across all agents/metrics is 15.41%; Program-of-Thought agent shows 21–25% gains.
• RL stage reduces token usage up to 66% relative to SFT alone while further improving accuracy.
• IMAD generalizes beyond arithmetic training domain (MMLU-Pro, BBH) without task-specific fine-tuning.

Limitations

• Training data is limited to six-operand arithmetic problems; multi-task extensions are explored only in the appendix.
• Fixed configuration of \(n=3\) agents and \(m=2\) rounds; scaling to more complex long-context tasks may require larger debate datasets.
• MMLU-Pro performance is not consistently improved: LLaMA IMAD scores 62.00% vs. SFT alone at 75.60%, indicating the RL internalization can hurt structured knowledge tasks.
• Malicious agent suppression results are truncated in the paper text; full quantitative comparison between IMAD and base model steering on evil/hallucination traits is not reproducible from the provided excerpt.
• Steering vector extraction is done from the SFT checkpoint (pre-RL) to avoid RL optimization artifacts, which may limit alignment between the steering vectors and the fully internalized model's representations.

Relevance to Agentic AI / LLM Agents

IMAD directly attacks a core scalability bottleneck of multi-agent systems by collapsing an \(n\)-agent debate protocol into a single forward pass, making agentic multi-perspective reasoning deployable in latency- and compute-constrained settings. The finding that agent-specific subspaces survive internalization suggests that collaborative reasoning structure is a learnable geometric property of LLMs, not merely an emergent behavior of multi-model orchestration — a result with implications for understanding how to build single-model agents that internally simulate diverse reasoning strategies. The behavioral control application (suppressing malicious internalized agents via negative steering) connects to alignment challenges in agentic deployments where role-playing or persona-conditioned agents may exhibit harmful behaviors. This work complements concurrent research on distilling multi-agent communication (Li et al., 2025; Luo et al., 2026) and advances the question of how far single-LLM architectures can absorb multi-agent capabilities.