Skip to content

Thucy: An LLM-based Multi-Agent System for Claim Verification across Relational Databases

🕒 Published (v1): 2025-12-02 22:35 UTC · Source: Arxiv · Venue: AAAI 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Thucy is a multi-agent LLM system that autonomously verifies natural-language claims against relational databases it has never seen before, returning both a verdict and the exact SQL queries that support it. It achieves 94.3% accuracy on TabFact, surpassing the prior state of the art by 5.6 percentage points. The system is the first to operate cross-table and cross-database while remaining fully source-agnostic at deployment time.

Problem

Existing LLM-based fact-verification systems are limited to single, small tables that fit within a context window, lack verifiable evidence trails, and require prior knowledge of the data schema. No prior system simultaneously handles cross-database queries, provides reproducible SQL evidence, and operates without pre-deployment knowledge of the data environment.

Method

Thucy uses a four-agent architecture built on the OpenAI Agents SDK. A Verifier (GPT-5) orchestrates three specialized expert-agents exposed to it as callable tools ("Agents as Tools" pattern): - Data Expert: performs a high-level scan of all connected databases, returning a single-paragraph summary so the Verifier can plan without context pollution. - Schema Expert: answers targeted schema questions using a context hint to avoid clutter from large or opaque schemas. - SQL Expert: iteratively writes, executes, and corrects SQL queries (NL2SQL loop), returning final answers with the exact supporting SQL.

Database connectivity is handled via Google's MCP Toolbox for Databases, which provides plug-and-play YAML-configured toolsets across PostgreSQL, MySQL, SQL Server, and Oracle. Expert agents are stateless across invocations, keeping them modular and reusable. The Verifier cycles through Data Expert → Schema Expert → SQL Expert as needed until it produces a four-class verdict (Verified / Partly Verified / Partly Inaccurate / Inaccurate) plus an explanatory SQL report.

Key Contributions

  • First cross-database, cross-table, source-agnostic claim verification system.
  • Verifiable verdicts via returned SQL queries that users can inspect, reproduce, and modify.
  • "Agents as Tools" orchestration pattern with stateless expert-agents, enabling cost-efficient model downgrading without significant accuracy loss.
  • New SOTA on TabFact small test set: 94.3% (GPT-5-mini experts) and 93.7% (GPT-4o-mini experts).
  • MCP-based plug-and-play data layer requiring only YAML config changes to add/remove databases.

Results

  • 94.3% accuracy on TabFact small test set (~2k pairs) with Verifier=GPT-5, experts=GPT-5-mini; +5.6 pp over previous best (AutoTQA at 88.7% with GPT-4-turbo).
  • 93.7% accuracy with experts downgraded to GPT-4o-mini; +5.0 pp over AutoTQA at same model tier.
  • Prior baselines: BINDER 85.1%, DATER 85.6%, CoTable 86.6%, ReAcTable 86.1%, AutoTQA 88.7%, POS 82.7%.
  • Real-world cost: ~\(0.05/example on TabFact; ~\)0.20/example for messy real-world claims; total TabFact evaluation ~$183.90.

Limitations

  • Benchmark mismatch: TabFact is single-table; no cross-database fact-verification benchmark exists, so the multi-database capability is not quantitatively evaluated.
  • Dirty data: missing values in schema columns (e.g., ~50% missing neighborhood field) can silently skew results; the system does not flag or handle this proactively.
  • Ambiguity handling: agents make implicit assumptions (e.g., current date, geographic definitions) that can introduce subtle errors without explicit grounding.
  • No ablation studies: individual agent contributions are not systematically isolated.
  • Stateless experts increase cost: agents re-discover schema information on each invocation, burning tokens redundantly across repeated calls.
  • High inference cost: multi-agent token consumption is substantial, limiting scalability without cost controls.

Relevance to Agentic AI / LLM Agents

Thucy is a concrete demonstration of the "Agents as Tools" orchestration pattern at production scale, where specialized stateless subagents are encapsulated behind well-defined interfaces and composed by a lightweight coordinator—directly validating design principles advocated in recent Anthropic multi-agent system work. The MCP-based tool layer shows how standardized protocols can decouple agent logic from database heterogeneity, a generalizable pattern for any agent system that must operate over unknown external data sources. The deliberate context management strategy—using cheap specialist agents to shield the expensive coordinator from noisy low-level data—is a transferable architectural insight for cost-efficient multi-agent design. The system also surfaces concrete failure modes (dirty data, ambiguous claims, stateless re-discovery cost) that are relevant to anyone building agents that ground reasoning in real-world structured data.