Skip to content

LegalHalluLens: Typed Hallucination Auditing and Calibrated Multi-Agent Debate for Trustworthy Legal AI

🕒 Published (v1): 2026-06-16 15:02 UTC · Source: Arxiv · Venue: ICML 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LegalHalluLens audits LLM hallucinations in legal contract extraction by profiling failures across four claim categories (numeric, temporal, obligation/entitlement, factual) and introducing a signed scalar directional metric. A six-role calibrated multi-agent debate pipeline—whose Skeptic challenges and asymmetric Add/Delete gates are derived from the measured failure profile—reduces fabricated detections by 45% and enables a 4B-active-parameter open model to match commercial APIs.

Problem

Aggregate hallucination rates (~52% across tested models) conceal a ~38–41 pp within-model gap between high-failure claim types (numeric, obligation) and low-failure types (temporal), providing compliance officers no actionable signal for procurement or deployment. Prior work establishes that legal hallucinations are non-uniform but neither operationalizes directional error character (omission vs. invention) as a deployment-comparable scalar nor addresses calibration of multi-agent mitigation to measured per-type failure modes.

Method

Typed hallucination profiles. All clause-level outputs on CUAD v1.0 (510 contracts, 41 annotated clause types → 249,252 instances) are stratified by claim category \(c_i \in \{\text{numeric, temporal, obligation, factual}\}\). Per-category \(\text{HalTP}(M, c_i)\) (content contradictions among detected clauses) is reported alongside the within-model gap \(\text{Gap}(M) = \max_{c_i} \text{HalTP}(M,c_i) - \min_{c_i} \text{HalTP}(M,c_i)\).

Risk Direction Index (RDI). The external judge labels each contradiction with a mismatch type including missing_condition (omission) and extra_condition (invention). RDI is defined as: $\(\text{RDI}(M) = \frac{p_\text{extra}(M) - p_\text{missing}(M)}{100}\)$ yielding a signed scalar: positive = invention-heavy; negative = omission-heavy. No extra annotation beyond the judge's existing labels is required.

Calibrated typed debate pipeline. A six-role state machine—Skeptic, Supporter, Re-extractor, Arbiter, Verifier, Judge—operates on a baseline extraction. Skeptic challenges are typed per claim category (e.g., for obligation: "Is the modal verb preserved? Are all carve-outs captured?"). An Addition Gate (absent→present) requires Verifier confirmation + debate consensus; a Deletion Gate (present→absent) is blocked when Verifier confirms presence, encoding the measured FAR > FRR asymmetry. A Re-extractor fires once on Round 1 structural errors. Maximum two debate rounds; Arbiter resolves deadlock under a conservative baseline-preserving policy.

Key Contributions

  • Typed hallucination profiles: consistent failure ordering \(\{\text{numeric, obligation}\} \gg \text{factual} \geq \text{temporal}\) across four architecturally diverse models, spanning ~38–41 pp gap invisible under aggregate reporting.
  • Risk Direction Index (RDI): a single signed scalar decomposing content errors into omission vs. invention, derived from judge labels already produced during evaluation, enabling directional comparison of systems with matched aggregate rates.
  • Calibrated multi-agent debate: Skeptic challenges and asymmetric Add/Delete gates derived from Experiment 1 diagnostics rather than generic tuning; claimed to be the first extraction pipeline whose agent design components are calibrated from measured per-failure-mode diagnostics.
  • Cost efficiency: gemma-4-26B-A4B (4B active MoE parameters) ranks first under 4 of 5 composite weighting schemes on the matched subset after typed debate, matching commercial APIs at substantially lower inference cost.

Results

  • Typed gap (Experiment 1, 249,252 instances): Numeric HalTP ranges 66.8–74.3%; obligation 64.8–73.6%; factual 36.0–46.9%; temporal 29.0–35.1% across all four models. Aggregate HalTP band: 50.9–56.5% (6 pp range, no actionable differentiation).
  • Within-model gap: 38.0–40.6 pp per model.
  • RDI differentiation: qwen3-32b (HalTP = 52.1%) RDI = −0.202 [−0.212, −0.193] (omission-heavy); gpt-5.2 (HalTP = 51.8%) RDI = +0.161 [+0.151, +0.170] (invention-heavy). Bootstrap CIs do not overlap.
  • Calibrated debate (Experiment 2, 120-contract subset, gemma-4-26B-A4B baseline): fabricated detections reduced by 45%; per-category gains concentrate on numeric and obligation as predicted; model ranks 1st under 4/5 weighting schemes on composite score vs. commercial API baselines.
  • Conservative abstention failure mode: llama-3.3-70b achieves lowest FAR (7.7%) and highest Acc (89.0%), yet numeric FRR = 52.8% and numeric JEq = 12.1% — fewer than 1 in 8 numeric clauses correctly extracted with correct content.

Limitations

  • Scope errors dominate contradictions (62–71%), compressing the directional variance captured by RDI — the metric captures only the subset of errors with clear omission/invention character.
  • Factual (n=3) and numeric (n=5) clause-type categories have small CUAD n; the central typed-gap claim rests on the obligation (n=27) vs. temporal (n=6) contrast.
  • Experiment 2 uses a single backbone model (gemma-4-26B-A4B) held out from Experiment 1; generalization of calibrated debate gains to other models is not demonstrated.
  • Single external evaluation judge (gemini-2.5-flash) introduces a single point of scoring bias; inter-annotator agreement with human experts is not reported.
  • Pipeline latency and per-query cost of six-agent debate vs. single-call baseline are not quantified.
  • Full-document context evaluation sets a performance ceiling; retrieval-augmented variants may compound additional errors not measured here.

Relevance to Agentic AI / LLM Agents

This work is a direct contribution to calibrated multi-agent pipeline design: it demonstrates that generic Skeptic prompts and symmetric gates are suboptimal, and that per-failure-mode diagnostics (typed rates + RDI) should function as first-class calibration inputs when assembling agentic extraction systems for high-stakes domains. The six-role debate state machine (Skeptic → Supporter → Re-extractor → Arbiter → Verifier → Judge with asymmetric gates) is a concrete architecture for error-correction agents where the FAR/FRR tradeoff is asymmetric and domain-measurable. For researchers building LLM agents in regulated domains, the RDI framework offers a transferable pattern: instrument the failure direction before designing agent roles, then let measured asymmetries drive gate thresholds rather than tuning them heuristically. The result that a 4B-active MoE model matches commercial APIs under typed debate directly motivates cost-efficient agent design via smaller specialized backbones augmented by structured debate.