Skip to content

Breaking Mental Set to Improve Reasoning through Diverse Multi-Agent Debate

šŸ•’ Published (v1): 2025-01-01 Ā· Source: ICLR Ā· Venue: ICLR 2025 Ā· link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

šŸ’¬ Ask ChatGPTāœ¦ Ask Claude

TL;DR

DMAD (Diverse Multi-Agent Debate) addresses the "mental set" problem in multi-agent LLM/MLLM debate by assigning each agent a structurally distinct prompting strategy rather than a shared one. Agents exchange full solving processes across rounds, cross-pollinating reasoning approaches to escape homogeneous failure modes. DMAD consistently outperforms standard MAD, self-reflection, and single-strategy baselines on both text-only and multimodal benchmarks.

Problem

Standard Multi-Agent Debate (MAD) assigns the same LLM multiple roles or personas but keeps all agents using the same underlying reasoning method (e.g., all CoT). This induces a "fixed mental set": when one reasoning approach fails, every agent fails together, and debate rounds cannot escape the shared blind spot. Self-reflection methods similarly cannot self-correct without external information injection.

Method

DMAD selects n maximally divergent prompting strategies R = {Rā‚, ā€¦, Rā‚™} and assigns each to a distinct agent instance of the same model. In round j, agent Mįµ¢ generates a solving process sįµ¢,ā±¼ and answer yįµ¢,ā±¼ using its assigned strategy Rįµ¢. All agents' (question, solving process, answer) tuples are pooled; each agent then receives the other agents' full solving processes and answers, updates its history, and produces a revised response in the next round. A judge Ļ• (self-consistency majority vote or model-selected best answer) extracts the final answer after each round.

For LLMs: R = {CoT, Step-Back Prompting, Program of Thoughts}. For MLLMs: R = {IO, Compositional CoT (CCoT, scene-graph-based), Duty-Distinct CoT (DDCoT, sub-question decomposition)}.

Key Contributions

  • Identifies "mental set" homogeneity as the root failure mode of existing MAD systems.
  • Proposes DMAD: a training-free framework assigning heterogeneous prompting strategies to debate agents using a single underlying model.
  • Demonstrates that sharing full solving processes (not just final answers) between agents is necessary for cross-strategy insight extraction.
  • Extends self-reflection evaluation to MLLMs, showing it performs even worse on MLLMs than LLMs (particularly small open-source models).
  • Provides systematic ablations over number of rounds and number of reasoning methods.

Results

  • MATH (GPT-4o-mini): DMAD 71.14% avg vs. best MAD (All CoT) 70.57%; outperforms all self-reflection baselines.
  • MATH (LLaMA-3-70B): DMAD 45.71% (second to MAD-All-PoT 46.43%, which dominates on calculation-heavy Number Theory); beats all non-PoT methods.
  • GPQA (GPT-4o-mini): DMAD 41.9% vs. MAD (All SBP) 40.2% ā€” best overall; similarly DMAD 41.1% tops all methods on LLaMA-3-70B.
  • ScienceQA: DMAD achieves 72.78% (LLaVA-1.6-13B), 85.57% (Gemini-1.5-Flash), 95% (GPT-4o).
  • MM-Vet: DMAD achieves 71.4% (Gemini, +1.1pp over MAD-All-DDCoT), 79.3% (GPT-4o, +1.5pp over MAD-All-DDCoT).
  • DMAD improves GPT-4o on MM-Vet by 7.3% and Gemini by 8.2% vs. baselines; standard MAD yields only 5.5% and 5.3%.
  • Efficiency: DMAD at 2 rounds surpasses MAD at 5 rounds on Gemini (ScienceQA, MM-Vet).
  • Performance scales monotonically with number of reasoning methods (n=1 < n=2 < n=3).

Limitations

  • Method relies on selecting sufficiently divergent prompting strategies; effectiveness degrades if chosen strategies are too similar.
  • When one strategy strongly dominates a benchmark type (e.g., PoT for LLaMA on calculation tasks), mixing it with weaker strategies can dilute performance rather than improve it.
  • Computational cost scales with nƗN agent calls per query; no cost analysis is provided.
  • LLaVA-1.6-13B with MAD series produces incoherent outputs on MM-Vet (long/Chinese-context questions), limiting MLLM generality to stronger models.
  • Evaluated on a fixed small set of strategy combinations; no principled strategy selection criterion is offered.

Relevance to Vision-Language Models

DMAD directly evaluates on three MLLMs (LLaVA-1.6-13B, Gemini-1.5-Flash, GPT-4o) using multimodal benchmarks (ScienceQA, MM-Vet), and introduces MLLM-specific debate strategiesā€”CCoT for visual scene understanding and DDCoT for text-centric sub-question decompositionā€”demonstrating that reasoning diversity is as critical in vision-language settings as in text-only ones. The paper is one of the first systematic studies showing that self-reflection techniques underperform even more severely on MLLMs than on LLMs, pointing to a structural gap in current VLM reasoning pipelines. For VLM researchers, DMAD offers a training-free, plug-and-play inference-time method to improve multimodal reasoning by exploiting the complementarity of image-focused vs. text-focused chain-of-thought prompting.