Skip to content

Who Checks the Citations? Benchmarking Legal Hallucination Detection

🕒 Published (v1): 2026-06-19 06:47 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Legal citation hallucinations in court filings are growing year-over-year despite model improvements, and no prior benchmark evaluated AI systems for detecting them. This paper introduces LePhantomCite, a 1,300-excerpt dataset with injected hallucinations across five taxonomic categories, and benchmarks agentic versus non-agentic LLMs on citation verification. The best agentic system (GPT-5 + BOED) reaches 82.8% recall / 60.5% F1; Claude Code with Opus 4.8 achieves the best precision (76.1%) and F1 (68.8%).

Problem

No benchmark existed for evaluating automated legal citation verification. Existing legal NLP work targets factual Q&A and summaries, not the specific failure modes (non-existent cases, name mismatches, wrong pincites, misquotes, content misrepresentation) that generate court sanctions. Meanwhile, hallucinated filing counts are growing and GPT-5.1 hallucinates at a significantly higher rate than the best mid-2024 GPT-4o (\(p = 0.001\)), falsifying the assumption that the problem is self-correcting.

Method

Dataset construction. 1,000 excerpts from pre-2022 federal appellate briefs (PDF→olmOCR→RoBERTa sentence segmenter→Llama-3.3-70B semantic grouper); 300 entries from the Large Legal Fiction dataset (Dahl et al., 2024) manually re-verified against Westlaw. Citation components (case name, quotation, holding) are extracted with Qwen3-32B constrained to exact substrings. Hallucinations are injected into 50% of segments at a single type per citation instance. Five types: non-existent citation, case name mismatch, incorrect pincite, verbatim misquote, content misrepresentation.

Agent harness. The BOED (Bayesian Optimal Experimental Design) framework from Zheng et al. (2026) is used. The agent maintains an explicit language-based belief state tracking every citation's status (correct / hallucinated / pending), updated after each of up to 30 actions. Eight actions include CourtListener search, local opinion exploration, and web search. Reflexion was trialed but abandoned because it loses track of citation validity decisions as action sequences grow long. The BOED belief state solves this by serving as persistent working memory.

Baselines. Gemini 2.5 Flash, GPT-5, GPT-OSS-120B, Qwen3-8B, Qwen3.5-27B — each in agentic (BOED, max_steps=30) and non-agentic (same prompt, no belief update) settings. Claude Code with Opus 4.8 (high effort, max_turns=30, WebSearch + WebFetch + CourtListener via MCP) is evaluated as a production-harness comparison.

Metric. Relaxed substring matching: a predicted segment and ground-truth segment match if one is a substring of the other. Standard precision, recall, \(\text{F1} = \frac{2 \cdot P \cdot R}{P + R}\) computed over 390 test excerpts.

Key Contributions

  • Longitudinal empirical evidence across 8 ChatGPT model generations that legal citation hallucination rates are not monotonically decreasing; GPT-5.1 rate (6.57%) significantly exceeds the best GPT-4o release (1.23%).
  • LePhantomCite: 1,300 excerpts, 4,499 citation instances, 1,107 hallucinated, grounded in a five-category taxonomy derived from real sanctioned court filings.
  • BOED harness evaluation showing agentic retrieval improves recall by 19.7% over non-agentic GPT-5; Claude Code (Opus 4.8) achieves the best precision (76.1%) and F1 (68.8%) by using tool calls more efficiently (~15.4 calls vs. GPT-5's average 16.9 steps).
  • Policy analysis: restricted access to commercial legal databases (Westlaw) is a structural bottleneck that disadvantages AI systems and unrepresented litigants equally.

Results

  • GPT-5 + BOED: 82.8% recall, 47.6% precision, 60.5% F1 (best recall among BOED-harness models).
  • Claude Code (Opus 4.8) native harness: 76.1% precision, 62.8% recall, 68.8% F1 (best precision and F1 overall).
  • Gemini 2.5 Flash + BOED: 65.3% recall, 20.2% precision, 30.9% F1.
  • GPT-OSS-120B + BOED: 57.7% recall, 24.5% precision, 34.4% F1.
  • Agentic vs. non-agentic improvement for GPT-5: recall +19.7 pp, F1 +9.4 pp.
  • Hardest categories: incorrect pincite (18.2% recall, GPT-5), verbatim misquote (82.6%), content misrepresentation (84.0%).
  • GPT-5 devotes 46.6% of actions to SEARCH_LOCAL_OPINION; averages 16.9 steps/excerpt.
  • Cumulative court filings with hallucinated citations exceeds 1,000 and is growing year-over-year.

Limitations

  • Dataset hallucinations are injected (50% positive rate), not naturally occurring; real-world rates are far lower and distribution differs.
  • Non-existent citation and case-name mismatch categories are relatively easy; pincite and misquote categories remain poorly solved even by the best models.
  • Information access asymmetry: CourtListener lacks pincite/page data; Westlaw access required for full verification but unavailable to public-facing agents.
  • Agentic verification is resource-intensive: 16.9 steps/excerpt at GPT-5, with max_steps=30 still insufficient for some excerpts.
  • Evaluation covers only U.S. federal appellate briefs (pre-2022); generalization to other jurisdictions, document types, or post-LLM filings is untested.
  • Content misrepresentation injection relies partly on LLM-generated examples (Dahl et al., 2024), with ground truth manually re-verified but potentially incomplete.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct benchmark of competing agent-harness designs applied to a high-stakes real-world task: the BOED harness (structured belief state + information-directed action loop) is pitted against the Reflexion framework (rejected), non-agentic prompting baselines, and Claude Code's native production harness. The comparison reveals concrete harness design trade-offs — BOED maximizes recall via exhaustive multi-step search; Claude Code's harness achieves higher precision and F1 by batching tool calls per turn (multiple calls per turn vs. one action per step), spending fewer total interactions. The belief-state mechanism is identified as the key component that makes agentic legal verification tractable, as it prevents citation-validity state from being lost across long action sequences. For researchers building meta-harnesses, the paper provides the first domain-specific evidence that harness architecture (step budget, belief representation, action granularity) materially affects end-task performance beyond the underlying model's capability.