Investigating Multi-Agent Deliberation in Law

🕒 Published (v1): 2026-06-29 20:56 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper evaluates multi-agent deliberation (MAD) frameworks for legal reasoning tasks, including a standard MAD setup and two novel law-inspired frameworks (3-Ply and Parrots). Multi-agent frameworks achieve comparable aggregate F1 to a single-LLM baseline but produce statistically distinct answers, successfully resolving cases the baseline fails on roughly half the time when they disagree.

Problem

Single-LLM approaches to legal reasoning produce one-sided narratives that risk tunnel vision on interpretation — a structural mismatch with law's inherently adversarial, multi-perspective deliberative nature. Multi-agent approaches to legal QA remain largely unexplored; prior work focused on simulation/argument generation rather than solving concrete legal reasoning benchmarks.

Method

Four frameworks are compared on 250 randomly-sampled binary (yes/no) questions each from five datasets (LEXAM, COLIEE, SARA, PRIVACY, LOGIQA; 1,250 total), using GPT-5-mini with 2-shot BM25-retrieved examples:

• Baseline: single model call with answer + short explanation.
• MAD: 3 agents each produce an initial prediction, then deliberate for 2 rounds (viewing each other's responses and revising); majority vote. Total: 9 model calls.
• 3-Ply: 3 agents play plaintiff (argues "yes"), defendant (argues "no"), and judge. Plaintiff makes an initial argument, defendant counterargues, plaintiff rebuts; judge decides. Total: 4 model calls.
• Parrots: A primary agent "Alex" makes an initial prediction, then argues back-and-forth with four critical "parrot" agents representing Socratic, Cynical, Eclectic, and Aristotelian perspectives (drawn verbatim from argumentation-theory literature). Alex can terminate early; max 3 rounds. Total: 3–7 model calls.

McNemar tests assess whether frameworks' decision patterns differ significantly from baseline.

Key Contributions

• Two novel law-inspired multi-agent deliberation frameworks (3-Ply and Parrots) applicable to binary legal reasoning tasks.
• Empirical evaluation across four legal benchmarks (LEXAM, COLIEE, SARA, PRIVACY) plus one logical reasoning benchmark (LOGIQA).
• Demonstration that multi-agent frameworks produce statistically distinct answers (\(p < 0.001\), McNemar) from baseline despite near-identical aggregate F1.
• Qualitative analysis showing structured deliberation reduces tunnel-vision errors on ambiguous/multi-interpretation legal clauses and produces richer, auditable justifications.
• Ablation tables for MAD agent count/rounds and Parrots static vs. dynamic round count on Privacy dataset.

Results

• Aggregate F1 scores across all five datasets differ by at most 1.5 points between any two frameworks; no framework consistently dominates.
• All three multi-agent frameworks disagree with baseline on ~7.5–10% of questions; of those disagreements, the multi-agent framework is correct ~47–52% of the time (3-Ply: 47.0%, MAD: 51.9%, Parrots: 42.6%).
• On Privacy, MAD F1 ranges from 88.99–90.31 across agent/round configurations (Table 1); Parrots F1 ranges 86.74–88.55 across round strategies (Table 2).
• Asymmetry observed on Privacy: every case the baseline solves is also solved by at least one multi-agent framework, but the reverse does not hold — multi-agent approaches surface reasoning inaccessible to the baseline.
• Qualitative: 3-Ply and Parrots more reliably navigate literal-vs.-pragmatic interpretation tradeoffs; MAD sometimes introduces hallucinated justifications before converging on the correct answer.

Limitations

• Experiments restricted to GPT-5-mini due to cost; larger models not explored.
• All tasks are binary (yes/no); frameworks not evaluated on open-ended or multi-class legal tasks.
• Only 250 questions per dataset (sampled); results may not generalize to full benchmark distributions.
• Hallucinations persist across all frameworks; MAD is particularly susceptible to agents importing irrelevant entities mid-deliberation.
• No standardized evaluation method for multi-interpretation legal reasoning exists; qualitative analysis is illustrative rather than systematic.
• Cost/compute overhead is significant: MAD requires 9× the calls of baseline, making deployment at scale expensive.

Relevance to Harnesses / Meta-Harnesses

The 3-Ply and Parrots frameworks are concrete examples of task-specific multi-agent harnesses: they wrap a base LLM with a fixed orchestration protocol (role assignment, turn sequencing, termination logic, aggregation) that structures deliberation without retraining the model. The variable call count in Parrots (agent-controlled early stopping) is a lightweight form of dynamic harness control — relevant to harness designers deciding how much to hard-code vs. delegate to the agent. The MAD ablation (Table 1–2) directly measures how harness parameters (number of agents, round count, static vs. dynamic termination) affect correctness and disagreement rate, which is exactly the kind of harness-configuration sensitivity analysis that meta-harness work needs. The finding that aggregate accuracy is flat while answer distributions are significantly different is a key empirical observation for meta-harnesses that aim to ensemble or route across agent configurations.