Skip to content

Debate or Vote: Which Yields Better Decisions in Multi-Agent Large Language Models?

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper disentangles Multi-Agent Debate (MAD) into majority voting and inter-agent debate components, finding empirically across seven benchmarks that majority voting alone accounts for most of MAD's performance gains. A formal Dirichlet-Compound-Multinomial (DCM) theoretical framework proves that debate induces a martingale over agents' belief trajectories, meaning debate does not improve expected correctness beyond the ensemble. Theory-informed interventions that bias belief updates toward correct signals yield meaningful improvements over vanilla MAD.

Problem

Multi-Agent Debate is widely used to enhance LLM reasoning, but the mechanisms driving its gains are unclear. Specifically, it is unknown whether performance improvements stem from inter-agent communication (debate) or simply from aggregating multiple independent outputs (ensembling via majority voting). This ambiguity makes it impossible to justify the added computational and architectural complexity of MAD over simpler baselines.

Method

The authors decompose MAD into two components and evaluate each independently: (1) Majority Voting (T=0 debate rounds, pure ensembling), and (2) Multi-Agent Debate (T=2,3,5 rounds with three communication topologies: Decentralized, Sparse, Centralized). Experiments use N=5 agents on Qwen2.5-7B, Llama3.1-8B, and Qwen2.5-32B across seven benchmarks.

For theory, each agent is modeled as a Dirichlet-Compound-Multinomial (DCM) process: a Dirichlet prior over answer beliefs is updated via Bayesian conjugacy as agents observe neighbor responses (count vectors). They prove that agents' belief in the correct answer forms a martingale (Theorem 2): E[p_{i,t} | α_{t-1}] = p_{i,t-1}, i.e., expected belief is unchanged by debate.

To improve MAD, two practical interventions are proposed guided by the martingale insight: - MAD-Conformist: agents that match the majority vote in round t retain their response in t+1. - MAD-Follower: with 30% probability, agents adopt the majority response; otherwise resample. An oracle variant (MAD-oracle) locks agents onto correct answers once found, establishing an upper bound.

Key Contributions

  • Comprehensive empirical demonstration across 7 benchmarks and 3 models that majority voting matches or surpasses MAD variants (Decentralized, Sparse, Centralized) at T=2,3,5 rounds.
  • Formal DCM-based theoretical framework: Theorem 1 gives a lower bound on majority voting success probability scaling with N; Theorem 2 proves debate is a martingale, providing a principled explanation for why debate adds no expected benefit.
  • Theory-informed MAD improvements (Conformist, Follower) that consistently outperform vanilla MAD by biasing updates toward majority-vote signals without oracle access.
  • Extension of findings to larger models (32B), heterogeneous persona agents, and open-ended summarization tasks (CNN/DailyMail, ROUGE metrics).

Results

  • Majority Voting vs. MAD (Qwen2.5-7B): Majority Voting average accuracy = 0.7691 vs. best MAD (Sparse T=2) = 0.7330; voting matches or beats all MAD variants on average across 7 benchmarks.
  • Majority Voting vs. MAD (Llama3.1-8B): Majority Voting average = 0.7242 vs. best MAD = 0.6990 (Sparse T=2); same pattern holds.
  • Larger model (Qwen2.5-32B, GSM8K): Majority Voting = 0.9433 vs. best MAD = 0.9400 (Decentralized T=2); majority voting remains competitive.
  • Improved MAD (Qwen2.5-7B, Decentralized T=2): MAD-vanilla avg = 0.7332; MAD-Conformist = 0.7625; MAD-Follower = 0.7629; MAD-oracle = 0.8236 — all exceed vanilla MAD.
  • Martingale empirical validation: Mean agent accuracy across debate rounds 1–5 is approximately flat for both models on all benchmarks, consistent with theoretical prediction.
  • Open-ended tasks (summarization): ROUGE-1 scores across T=0–3 rounds range 0.2686–0.2825, confirming debate adds minimal gain.

Limitations

  • The DCM model is an idealized abstraction of LLM generation; real LLMs with chain-of-thought, instruction tuning, or RLHF may deviate from Dirichlet-Multinomial dynamics.
  • MAD-Conformist and MAD-Follower rely on majority vote as a proxy for correctness, which can fail when the majority is wrong (common misconception regime).
  • Heterogeneous persona experiments show some MAD variants outperform majority voting on MMLU Pro. Med., suggesting persona diversity may matter in specific task-model combinations, a case not fully explained by the theory.
  • Evaluation is limited to N=5 agents; scaling behavior at very large N is analyzed theoretically but not extensively empirically.
  • Open-ended evaluation uses only one summarization dataset and a best-agent proxy rather than a true voting aggregation mechanism.

Relevance to Agentic AI / LLM Agents

This paper directly challenges a foundational assumption behind multi-agent LLM systems: that inter-agent deliberation is the key driver of performance. For practitioners building agentic pipelines, the finding that majority voting over independent runs replicates most of MAD's gains substantially changes the cost-benefit calculus of complex debate architectures. The martingale result provides a rigorous theoretical caution against designing agent loops that iterate without a mechanism to preferentially preserve correct reasoning, which is actionable guidance for system designers. The MAD-Conformist/Follower interventions show that injecting structured consensus signals (majority vote) into individual agent updates is a lightweight path to real debate benefit, pointing toward hybrid voting-debate architectures rather than pure debate.