iMAD: Intelligent Multi-Agent Debate for Efficient and Accurate LLM Inference

🕒 Published (v1): 2025-11-14 13:50 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

iMAD is a selective Multi-Agent Debate (MAD) framework that avoids triggering expensive multi-agent debate for every query, instead using a lightweight MLP classifier fed with 41 linguistic/semantic features from a structured self-critique response to decide when debate is likely to correct a wrong answer. On six QA/VQA benchmarks, iMAD reduces token usage by up to 92% versus full MAD while improving accuracy by up to 13.5% over single-agent baselines.

Problem

Standard MAD frameworks trigger debate on every query, incurring 3–5× more tokens than single-agent CoT. Worse, debate is often harmful or redundant: only 4.9–19.1% of cases are the "correctable error" scenario where debate helps; in many others it flips correct answers wrong. Confidence-score-based selective triggering (e.g., DOWN) is unreliable because LLMs are frequently overconfident on incorrect answers, and it requires labeled evaluation data to tune thresholds, violating zero-shot assumptions.

Method

iMAD operates in three stages: 1. Structured self-critique prompting: A single LLM agent is prompted to produce (i) an initial CoT justification, (ii) a required counterargument, and (iii) explicit confidence scores for both perspectives — eliciting hesitation cues without additional input tokens. 2. Feature extraction: 41 interpretable linguistic and semantic features are extracted from the structured output, covering surface statistics, readability scores, POS counts, question-type indicators, and lexical hedging/contrast cues. 3. Debate decision classifier: A 6-layer MLP (200 hidden units) takes the feature vector (including raw LLM confidence score) and outputs a score p. If p < threshold τ=0.7, MAD is triggered; otherwise the single-agent answer is kept. The classifier is trained with the proposed FocusCal loss (L_FC), a composite of: (a) Asymmetric Focal Loss (penalizes high p on incorrect answers to combat overconfidence), (b) Confidence Penalty (aligns predicted score p with an auxiliary uncertainty head trained on semantic hesitation features), and (c) Expected Calibration Error (aligns p with empirical correctness). The classifier is trained only on PubMedQA and GQA, then applied zero-shot across all six evaluation datasets.

Key Contributions

• iMAD framework: selective MAD triggering via structured self-critique + lightweight MLP classifier, requiring no evaluation data.
• 41 interpretable linguistic/semantic features that expose hesitation cues better than raw LLM confidence scores.
• FocusCal loss combining asymmetric focal loss, confidence penalty, and ECE regularization to learn generalizable debate-triggering behavior.
• Empirical analysis (Insights 1–3) quantifying the token overhead of MAD, the breakdown of beneficial vs. harmful debate outcomes, and the unreliability of confidence scores as debate triggers.

Results

• Token reduction vs. full MAD: up to 92% fewer tokens (vs. GroupDebate on MEDQA: 16,832 → 1,300).
• Accuracy vs. single-agent CoT: up to +13.5% (GSM8K: 71.3% → 84.8%).
• Accuracy vs. full MAD (Liang et al.): iMAD matches or exceeds on 5/6 datasets; e.g., MEDQA 82.0% vs. MAD 81.9% at 68% fewer tokens; GSM8K 84.8% vs. MAD 76.4%.
• vs. Self-Consistency (SC): iMAD reduces tokens by 62.7% on MEDQA while gaining +4.7% accuracy.
• vs. DOWN (confidence-based selective): comparable token cost but consistently higher accuracy; e.g., OKVQA DOWN=88.1% vs. iMAD=90.3%.
• Decision quality: up to 95.9% of iMAD's debate decisions are beneficial; harmful flips (✓→✗ when triggered) remain ~5–10% across datasets.

Limitations

• Classifier requires a labeled training set (PubMedQA + GQA), introducing some supervision cost even if held out from evaluation sets.
• Performance degrades on MMLU (short, factual, cross-domain questions) where incorrect answers provide few hesitation cues — MAD slightly outperforms iMAD there.
• Self-critique adds a small but non-zero output-token overhead versus bare CoT or confidence-only baselines like DOWN.
• Evaluated with Gemini 2.0 Flash as the primary LLM; generalization to other model families (GPT-5 nano, Qwen 3.0) is reported in appendices but not the main results table.
• Confidently wrong single-agent answers (coherent but flawed reasoning) remain largely unrecoverable by MAD — iMAD learns to skip these but cannot fix them.

Relevance to Agentic AI / LLM Agents

iMAD directly addresses the compute efficiency bottleneck of multi-agent debate systems, a core challenge in deploying LLM agent pipelines at scale. The selective triggering paradigm — deciding when to escalate from single-agent to multi-agent deliberation — is broadly applicable to agentic architectures that currently invoke expensive collaborative reasoning indiscriminately. The work also provides a principled empirical decomposition of when MAD actually helps versus hurts, offering a useful diagnostic framework for anyone designing multi-agent reasoning pipelines. The FocusCal loss and hesitation-feature approach contribute reusable mechanisms for uncertainty-aware routing in agent systems.