Skip to content

MAD-Logic: Multi-Agent Debate Enhances Symbolic Translation and Reasoning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Multi-agent debate improves symbolic translation and logical reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MAD-Logic introduces a multi-agent debate framework for logical question answering that simultaneously addresses symbolic translation errors and reasoning failures by running agents over multiple symbolic languages (LP, FOL, SAT) and natural language, debating across both stages, and using adaptive sparse communication to prune redundant agent interactions.

Problem

Existing logical QA methods face a fundamental trade-off: symbolic language (SL) solver pipelines (LP/FOL/SAT + Pyke/Prover9/Z3) achieve rigorous reasoning but are brittle to translation errors, while NL-based prompting methods (CoT, Plan-and-Solve) tolerate translation noise but suffer from hallucinations. No prior work jointly addresses translation quality and reasoning accuracy via multi-agent debate, and all prior multi-agent debate methods incur prohibitive token costs from dense communication.

Method

MAD-Logic operates in two debated stages:

Translation stage: Three agents translate the NL problem in parallel into LP, FOL, and SAT. Agents then debate for \(D\) rounds (optimal \(D=2\)–\(3\)) to mutually correct translation errors via cross-refinement, maximizing solver execution rates.

Reasoning stage: Solver-based agents (Pyke for LP, Prover9 for FOL, Z3 for SAT) produce symbolic reasoning traces verbalized as natural language narratives; NL agents (CoT, Plan-and-Solve) produce their own traces. All agents debate for \(N\) rounds, each rewriting its reasoning narrative given the others' outputs. Final answer: majority vote over all agents' conclusions.

Sparse communication: Agent \(i\)'s output is forwarded to agent \(j\) at round \(d\) only if the preference score \(\text{Pred}_{i \to j} = C_i^d / C_j^d + \lambda(1 - \cos(A_j^d, A_i^d \| A_j^d))\) exceeds the historical average \(\text{Pred}_{i \to j}^{d-1}\) scaled by threshold \(\alpha\). The score combines confidence ratio (high-confidence source to low-confidence receiver) with information gain (cosine distance of outputs). Each agent maintains a personalized memory updated only from open communication channels.

Key Contributions

  • First multi-agent debate framework that spans both NL→SL translation and hybrid SL/NL reasoning stages jointly.
  • Adaptive sparse communication gate using confidence ratio and information gain, reducing token consumption while empirically improving accuracy over fully-connected debate.
  • Theoretical lower bound on majority-vote ensemble accuracy showing convergence to 1 when pairwise agent error correlation \(\rho = 0\), and graceful degradation at low \(\rho > 0\).
  • Empirical evaluation across 6 benchmarks (3 synthetic: ProntoQA, ProofWriter, LogicalDeduction; 3 real-world: AR-LSAT, FOLIO, Chinese LogiQA-V2) with GPT-4, Claude 3.7 Sonnet, DeepSeek-V3, and Qwen2.5-7B.

Results

  • Synthetic benchmarks (GPT-4): MAD-Logic (w/ sparse) reaches 100% / 92.00% / 94.33% on ProntoQA / ProofWriter / LogiDeduct, vs. best prior multi-agent baseline CortexDebate at 99.60% / 90.83% / 92.33%.
  • Synthetic benchmarks (DeepSeek-V3): 100% / 93.33% / 100% vs. CortexDebate's 99.80% / 93.00% / 99.67%.
  • Real-world benchmarks (GPT-4): 53.25% / 86.27% / 74.76% on AR-LSAT / FOLIO / Chinese LogiQA-V2 vs. CortexDebate's 51.08% / 84.80% / 74.13%.
  • Real-world benchmarks (DeepSeek-V3): 75.76% / 90.67% / 86.93% vs. CortexDebate's 74.03% / 88.73% / 83.04%.
  • Smaller model (Qwen2.5-7B): w/ sparse achieves 86.40% / 76.50% / 67.67% on ProntoQA / ProofWriter / LogiDeduct vs. CortexDebate (w/ NL-SL) at 84.00% / 75.83% / 67.00%.
  • Sparse vs. full debate: Sparse variant consistently matches or exceeds fully-connected variant while saving 13–36% tokens (Table 7, \(\lambda=1.0\), \(\alpha\) optimal).
  • Statistical significance (Table 4): w/ sparse improvements over CortexDebate are statistically significant (\(p < 0.05\)) on most benchmarks for GPT-4 and DeepSeek-V3.
  • Ablation: Removing symbolic reasoning agents causes the largest accuracy drop; removing translation debate causes the second largest.

Limitations

  • Translation debate quality degrades beyond 2–3 rounds due to over-correction noise, requiring careful tuning of \(D\).
  • Performance on AR-LSAT remains low (53% with GPT-4, 37% with Qwen2.5-7B), suggesting hard real-world logical puzzles remain partially unsolved.
  • Sparse gate introduces two additional hyperparameters (\(\alpha\), \(\lambda\)) with non-trivial sensitivity (Table 7 shows accuracy peaks at \(\lambda=1.0\)–\(1.5\) before declining).
  • Evaluation is restricted to closed-world logical QA (true/false/unknown); generalization to open-domain or multi-hop reasoning is not demonstrated.
  • Token savings (13–36%) are meaningful but the baseline MAD-Logic still incurs higher cost than single-agent methods; no latency wall-clock measurements are reported.

Relevance to Agentic AI / LLM Agents

MAD-Logic is a direct example of heterogeneous multi-agent debate as a reasoning primitive: agents differ not only in LLM backbone but in their representational modality (LP vs. FOL vs. SAT vs. NL), making the ensemble qualitatively more diverse than homogeneous LLM panels. The adaptive sparse communication mechanism—gating inter-agent messages by confidence ratio and information gain rather than broadcasting to all—is a practical efficiency technique applicable to any multi-agent pipeline where token cost scales quadratically with agent count. The theoretical analysis under majority-vote ensemble provides a formal grounding for when heterogeneous agent pools outperform single powerful agents, which is broadly relevant to agentic orchestration design. The hybrid NL/SL architecture also illustrates how tool-augmented agents (symbolic solvers) and pure-LLM agents can be composed in a single debate loop to achieve complementary coverage.