Skip to content

Courtroom-Style Multi-Agent Debate with Progressive RAG and Role-Switching for Controversial Claim Verification

🕒 Published (v1): 2026-03-30 14:23 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PROCLAIM reformulates fact-checking as a structured legal proceeding: specialized LLM agents (Plaintiff, Defense, Judge, Critic, Expert Witness) debate claims through iterative, adversarially-balanced evidence retrieval (Progressive RAG) with role-switching consistency tests and heterogeneous multi-judge aggregation. On the Check-COVID benchmark, the full pipeline achieves 81.7% majority-vote accuracy, a +10.0 pp gain over standard multi-agent debate, with P-RAG alone accounting for +7.5 pp.

Problem

Standard RAG pipelines use static single-pass retrieval, leaving adversarial evidential gaps unexplored as debate evolves. Unstructured multi-agent debate (MAD) causes premature convergence, shared-bias reinforcement (epistemic bubbles), and agent agreement being misread as correctness — all critical failures in high-stakes claim verification.

Method

PROCLAIM (Progressive Retrieval Orchestrated multi-agent framework for Claim verification) operates in four layers:

  1. Argument Mining: An LLM decomposes the raw claim \(c\) into atomic premises \(P = \{p_1,\ldots,p_k\}\) that seed retrieval and serve as a checklist for critic evaluation.

  2. Evidence Negotiation: Stance-conditioned queries (one proponent, one opponent) are issued to a FAISS index over ~360K PubMed COVID abstracts (all-MiniLM-L6-v2 embeddings, 384-dim). Evidence is scored by admissibility weight \(w = r \times c\) (relevance × credibility); items with \(w > 0.5\) are admitted, \(0.1 < w \leq 0.5\) flagged, \(w \leq 0.1\) discarded.

  3. Progressive RAG (P-RAG): Each debate round, a Judge-refined query is formed from (i) rolling debate context (last 4 messages), (ii) the agent's self-identified evidential gap, and (iii) reflection-driven discovery needs. A novelty filter rejects candidates where \(\text{novelty}(d) = 1 - \max_{p\in P}\cos(e_d, e_p) < 0.20\). Retrieval stops when redundancy ratio >70%, relevance gain <0.05, or after 10 iterations.

  4. Debate Orchestration: Five roles across heterogeneous models (deepseek-r1, gpt-5-mini, deepseek-v3.2, qwen3-235b, hermes-3-llama-405b) execute up to 10 rounds. Each round: (1) P-RAG evidence discovery, (2) argument generation, (3) optional expert witness testimony, (4) per-agent self-reflection scored as \(s_{\text{ref}} = 0.4l + 0.3n + 0.3b\) (logic, novelty, rebuttal), (5) Critic Agent evaluation and early-stop signal.

  5. Role-Switching Consistency Test: After the primary debate, Plaintiff and Defense swap and re-run the full debate; a separate LLM scores consistency between the two transcripts, producing \(\delta_{\text{rs}}\).

  6. Judicial Panel: Three heterogeneous judges independently score across six dimensions; majority vote determines the verdict. Final confidence: \(c_{\text{final}} = \text{clamp}(c_{\text{base}} + \delta_{\text{rs}} + \delta_{\text{ref}}, 0, 1)\) where \(c_{\text{base}} = 0.8\sigma + 0.3q\) (\(\sigma\) = consensus strength, \(q\) = normalized judge quality). Weights calibrated by 5-fold CV grid search minimizing ECE on a held-out dev subset.

Key Contributions

  • Courtroom-structured debate pipeline (PROCLAIM) with explicit adversarial roles and evidence admission protocols
  • P-RAG: iterative, gap-driven retrieval with novelty filtering and adaptive stopping, functioning as the primary performance driver (+7.5 pp)
  • Role-switching as a consistency diagnostic (not just bias mitigation): evidence-grounded vs. position-anchored reasoning
  • Heterogeneous multi-judge panel: model diversity yields complementary error profiles, +3.3 pp over single-judge panels
  • Self-reflection as an economic governor: reduces debate rounds by 29% (7.06→5.47) and token usage by 17% at <1 pp accuracy cost
  • Empirical characterization of LLM reasoning pathologies: epistemic bubbles (removing P-RAG raises \(\bar{\kappa}\) from 0.468→0.599 while dropping accuracy 7.5 pp), structural negativity bias (LLMs over-produce REFUTE, converging 0.2–0.3 rounds faster), and butterfly effect of premise framing (stochastic decomposition in Run-2 drops \(\kappa_{\text{GT}}\) to 0.384 despite stable inter-judge \(\kappa\))

Results

All results on Check-COVID binary subset (SUPPORT/REFUTE); 360 debate instances total, ablations on 120 claims.

  • PROCLAIM (majority vote): 81.7% accuracy, 81.7% macro-F1, \(\bar{\kappa}=0.468\), avg 5.47 rounds, avg 67.5 evidence items, 210.9K tokens
  • vs. Standard MAD: +10.0 pp accuracy (71.7% → 81.7%)
  • Oracle ceiling: 95.8% (Run-0 alone: 95.0%)
  • vs. Single-call GPT-5-mini + RAG: −4.2 pp (85.8%), but no deliberative traceability or adversarial robustness
  • vs. Single-call DeepSeek-v3.2 + RAG: +1.7 pp (80.0%)
  • Ablation (−P-RAG): −7.5 pp, evidence pool halved (37.5 vs. 67.5), \(\bar{\kappa}\) rises to 0.599
  • Ablation (−Role-Switching): −4.2 pp, evidence pool −20%
  • Ablation (−Three-Judge Panel): −3.3 pp
  • Ablation (−Self-Reflection): −0.8 pp, rounds increase to 7.06, tokens increase by 17%
  • Confidence calibration: ECE = 0.034 (5× improvement over standard averaging ECE = 0.18)

Limitations

  • Evaluated on a single domain (COVID-19 biomedical claims from PubMed); generalizability to other domains not demonstrated
  • Stochastic variability across runs is substantial (Run-0: 95.0%, Run-2: 79.0%); upstream premise decomposition can propagate systematic bias through the entire pipeline
  • High token cost: ~210.9K tokens per claim at full pipeline; unsuitable for high-throughput or latency-sensitive applications
  • Monolithic single-call baselines (GPT-5-mini + RAG at 85.8%) outperform PROCLAIM on point accuracy, trading deliberative traceability for raw performance
  • Evidence corpus is static (PubMed, 2020–2024); dynamic or open-web retrieval not evaluated
  • Role-switching doubles compute per claim; the consistency score's marginal accuracy contribution (\(\delta_{\text{rs}}\)) is not isolated in ablations

Relevance to Harnesses / Meta-Harnesses

PROCLAIM is a direct instance of a multi-agent orchestration harness: a fixed pipeline that sequences specialized sub-agents (miners, counsels, witnesses, critics, judges) with structured inter-agent communication protocols, adaptive stopping conditions, and feedback loops between modules. The P-RAG mechanism — where each agent's self-reflection output feeds the next retrieval query — is architecturally equivalent to a meta-harness feedback loop that dynamically steers sub-agent behavior based on prior outputs. The ablation study provides quantitative evidence that harness structure (which agents run, in which order, with which stopping criteria) dominates model choice, a key result for researchers designing multi-agent orchestration systems.