Skip to content

EMULATE: A Multi-Agent Framework for Determining the Veracity of Atomic Claims by Emulating Human Actions

🕒 Published (v1): 2025-05-22 12:08 UTC · Source: Arxiv · Venue: ACL 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

EMULATE is a multi-agent fact-checking framework that verifies atomic claims by decomposing the human web-search-and-verify process into seven specialized LLM agents. Each agent handles a discrete subtask (query generation, result ranking, self-containedness checking, helpfulness gating, evidence sufficiency, classification, and follow-up query generation). It outperforms four baselines across three benchmarks on 6 of 8 reported metrics.

Problem

Existing atomic claim verification systems either (a) retrieve evidence once and classify in a single pass, misaligning with how humans iteratively gather evidence, or (b) use coarse iterative retrieval (e.g., FIRE) without explicitly reasoning about why additional evidence is needed or filtering redundant/irrelevant results before they enter the evidence set.

Method

EMULATE pipelines seven zero-shot LLM agents over a bounded search loop (\(\text{MAX\_SEARCH\_QUERIES}=4\), \(\text{MAX\_SEARCH\_RESULTS\_PER\_QUERY}=2\)):

  1. InitialQueryGen — generates search queries from the claim.
  2. SearchRank — sorts results by source credibility (URL) and snippet relevance.
  3. SelfContainedCheck — determines whether a retrieved document is comprehensible given the current evidence bank.
  4. DetHelpful — gates admission: accepts a result only if it provides novel, task-relevant information.
  5. SufficientEvidence — decides whether to terminate retrieval or continue.
  6. AdditionalQueryGen — generates targeted follow-up queries when more evidence is needed.
  7. Classifier — outputs a binary True/False verdict over the accumulated evidence set.

Documents flagged as non-self-contained are deferred to a post-loop pass rather than triggering immediate additional queries, prioritizing self-contained evidence to minimize query count.

Key Contributions

  • A seven-agent decomposition of the human fact-checking process, including explicit credibility/relevance ranking and self-containedness filtering not present in prior systems.
  • Iterative, need-driven evidence accumulation with an explicit sufficiency gate, extending FIRE's iterative retrieval with finer-grained control.
  • Demonstrated state-of-the-art performance on FacTool-KBQA, BingCheck, and Factcheck-Bench under a unified GPT-4.1 backbone.
  • Ablation evidence that SearchRank and SelfContainedCheck each individually contribute measurable gains.
  • Robustness analysis showing near-competitive performance with GPT-4.1-mini vs. GPT-4.1.

Results

All numbers use GPT-4.1 at temperature 1; baselines are FacTool, FactCheck-GPT, SAFE, and FIRE.

  • BingCheck: EMULATE Macro-F1 = 0.81 vs. FIRE 0.76 (best prior); False-class F1 = 0.69 vs. FIRE 0.63.
  • FacTool-KBQA: EMULATE Macro-F1 = 0.80 vs. FIRE 0.78; False-class F1 = 0.68 vs. FIRE 0.66; True-class F1 = 0.91 vs. FIRE 0.89.
  • Factcheck-Bench: EMULATE Macro-F1 = 0.80 vs. FIRE 0.78; False-class F1 = 0.71 vs. FIRE 0.68; True-class F1 = 0.90 vs. FIRE 0.87.
  • EMULATE achieves best results on 6/8 metrics across all three datasets; FIRE is consistently second-best.
  • Ablation: removing SearchRank drops False F1 by 0.11 on FacTool-KBQA (0.68→0.57); removing SelfContainedCheck drops False F1 by 0.05 (0.68→0.63).

Limitations

  • Evaluation limited to three small datasets (all under 1,000 examples) due to scarcity of atomic-claim-level veracity annotations.
  • Class imbalance (True >> False) in all datasets may inflate True-class metrics and obscure False-class weaknesses.
  • Non-self-contained documents are deferred to end-of-loop processing; for some claims this heuristic may be suboptimal if critical evidence only appears in non-self-contained sources.
  • No evaluation on multilingual or non-English claims; web search via serper.dev introduces external API dependency.

Relevance to Harnesses / Meta-Harnesses

EMULATE is a textbook example of a task-decomposition harness: it wraps a complex multi-step reasoning pipeline (web retrieval → ranking → filtering → sufficiency checking → classification) inside a coordinating loop that dispatches specialized agents, controls flow between them, and enforces termination conditions. The explicit sufficiency gate and deferred-document queue are harness-level control structures, not model behaviors, demonstrating how a meta-harness can compensate for LLM limitations (ignoring irrelevant context, knowing when to stop) through architectural design rather than prompting alone. For researchers building harnesses, the ablation results directly quantify the contribution of individual harness components (ranking and filtering agents), offering a rare empirical decomposition of harness-component value.