Skip to content

Source or It Didn't Happen: A Multi-Agent Framework for Citation Hallucination Detection

🕒 Published (v1): 2026-05-09 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CiteTracer reframes citation hallucination detection as taxonomy-aligned, field-level adjudication using a 12-code taxonomy (REAL, POTENTIAL, HALLUCINATED) and a four-stage cascading multi-agent pipeline. It achieves 97.1% accuracy on a 2,450-citation synthetic benchmark and 97.1% recall on 957 real-world fabricated citations from ICLR 2026 desk-rejections, surpassing all five frontier baselines under both PDF and BibTeX input modes.

Problem

Existing citation auditors (Citely, CiteAudit, Hallucinator, GPTZero) reduce verification to binary REAL/FAKE labels, obscuring which bibliographic field (title, author, venue, year, identifier) is wrong and collapsing the auditor-ambiguous middle ground into a single signal. Additionally, brittle PDF parsing corrupts field extraction before any verification occurs, and no tool covers the full field surface (venue, DOI, pages, publisher, peripheral metadata) while also supporting non-BibTeX PDF input at scale.

Method

CiteTracer is an end-to-end agentic pipeline \(\text{CiteTracer}(P) = \{(r_i, y_i, \Delta_i, S_i)\}_{i=1}^{N}\) with four sequential modules:

  1. Reference Extractor: An OCR model acts as a high-recall citation-block proposer; a vision-LLM parsing agent \(A_{\text{Parser}}\) then re-parses each cropped block image alongside the OCR transcription, producing structured field records \(r_k = A_{\text{Parser}}(P[B_k], T_k)\) and handling cross-column/cross-page boundary merging.

  2. Cascading Evidence Collector: A four-stage cascade ordered cheapest-to-most-general—(i) Memory (DBLP mirror cache), (ii) URL Fetch (DOI/arXiv direct), (iii) parallel Scholar Connectors (8 bibliographic APIs), (iv) Web Search—with early exit on the first stage that supports a REAL verdict, building cumulative evidence bundle \(E_i\).

  3. Field Matcher: A two-stage matcher first applies deterministic field-specific normalizers (case folding, DOI canonicalization, venue abbreviation lookup) with early exit when \(\exists e \in E_i: \forall f \in F_i^+, m^{\text{rule}}_{i,e,f} = \text{MATCH}\); unresolved cases proceed to \(A_{\text{Matcher}}\) which produces a residual field-status profile \(m_i\) labeling reorderings, missing metadata, or true contradictions.

  4. Class-Specialist Judgers: A routing function \(\rho(r_i, E_i, m_i)\) directs each case to one of three specialists (Valid, Potential, Hallucinated Judger), each applying class-appropriate evidence thresholds to emit \((y_i, \Delta_i, S_i)\)—the taxonomy code, offending field spans, and supporting sources.

Key Contributions

  • 12-code taxonomy: R1–R3 (exact match, format variant, et al.), P1–P3 (nickname variant, non-academic source, unverifiable peripheral), H1–H6 (title, author, venue, year, identifier, peripheral errors), giving per-field diagnostic granularity.
  • Dual benchmark: 2,450-citation synthetic set (constructed from DBLP/arXiv/ACL seeds with LLM-driven code-specific mutations, triple-checked) across five bibliography styles; plus a 957-citation real-world set from ICLR 2026 and one anonymous conference desk-reject queues.
  • CiteTracer framework: Verdict-driven cascading retrieval with early exit, deterministic field matching with LLM escalation, and class-specialist judger routing—together avoiding wasteful LLM calls on easy majority cases.
  • Open benchmark and code released at github.com/aaFrostnova/CiteTracer.

Results

  • Synthetic benchmark (BibTeX): 97.1% overall accuracy; class-level F1: REAL 97.0, POTENTIAL 95.8, HALLUCINATED 98.5.
  • Synthetic benchmark (PDF): F1: REAL 95.1, POTENTIAL 95.5, HALLUCINATED 96.9.
  • Real-world set: 97.1% recall on 957 fabricated citations without abstaining.
  • vs. GPT-5.5 (BibTeX): REAL F1 94.1→97.0, POTENTIAL F1 43.8→95.8, HALLUCINATED F1 89.2→98.5.
  • vs. Claude 4.7 Opus (BibTeX): REAL F1 87.5→97.0, POTENTIAL F1 35.6→95.8, HALLUCINATED F1 84.6→98.5.
  • vs. Gemini 3.1 Pro (BibTeX): POTENTIAL F1 14.9→95.8, HALLUCINATED F1 61.8→98.5.
  • Hallucinator (binary, title+author only): REAL F1 63.1 / HALLUCINATED 67.3 on PDF.
  • GPTZero (PDF): HALLUCINATED TPR 45.9%, FPR 12.0%.
  • CiteTracer subtype TPRs (BibTeX): H1–H6 all ≥95.4% except H6 (100.0%), P1 100.0%, P3 99.4%.

Limitations

  • P2 (non-academic sources: blogs, GitHub, forums) is excluded from the synthetic benchmark as a routing case rather than a verification case; its detection performance is not quantified.
  • 58 citations were excluded from the PDF evaluation due to rendering artifacts in the benchmark PDFs.
  • The real-world test set is entirely HALLUCINATED by construction (chairs' desk-reject queue), so real-world REAL/POTENTIAL precision cannot be measured.
  • Retrieval coverage depends on public bibliographic APIs; domain-specific or paywalled venues may cause the cascade to fall through to Web Search with lower precision.
  • Benchmark mutations are LLM-generated; distribution shift between synthetic mutation patterns and organic LLM hallucination styles is not directly measured.

Relevance to Harnesses / Meta-Harnesses

CiteTracer is a textbook implementation of a multi-agent verification harness: it decomposes a complex judgment task into a strict sequence of specialized agents (extractor → retrieval cascade → deterministic matcher → specialist judgers), each with a clearly bounded scope and handoff contract. The verdict-driven early-exit pattern in the evidence cascade is a concrete cost-control primitive—the harness routes cheap cases through cheap stages and reserves expensive LLM calls only for the residual ambiguous tail—which is directly generalizable to other multi-step verification pipelines. The class-specialist routing architecture (rather than a monolithic judge) demonstrates how harness design can improve calibration by matching decision logic to error taxonomy, a transferable principle for any pipeline where failure modes are structurally heterogeneous. For meta-harness researchers, this paper provides an end-to-end existence proof that cascade-with-early-exit + specialist routing outperforms frontier monolithic LLMs by large margins on a structured verification task.