Skip to content

FACTS: Table Summarization via Offline Template Generation with Agentic Workflows

🕒 Published (v1): 2025-10-15 10:24 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FACTS is an agentic workflow for query-focused table summarization that generates reusable offline templates—pairs of SQL queries and Jinja2 rendering templates—from table schemas alone, never exposing raw row data to LLMs. A multi-LLM ensemble ("LLM Council") iteratively validates each intermediate artifact via majority vote and aggregated feedback. FACTS outperforms prompt-based and prior agentic baselines on FeTaQA, QTSumm, and QFMTS.

Problem

Existing approaches to query-focused table summarization each fail at least one of four practical requirements: (1) speed/reusability — prompt-based methods re-query the LLM for every new table; (2) accuracy — free-form generation produces hallucinated values; (3) privacy compliance — sending raw table rows to external LLMs violates HIPAA/GDPR; (4) scalability — feeding large tables exhausts token limits. Prior agentic frameworks (Binder, Dater, TaPERA, SPaGe) improve accuracy but sacrifice the other three.

Method

FACTS runs three sequential stages, each guarded by an LLM Council:

  1. Schema-Guided Specification (Stage 1): Given the user query and table schema (no raw values), the agent generates (i) guided questions identifying relevant columns/relations and (ii) abstract filtering rules (e.g., exclude category='expense'). The Council votes YES/NO with feedback; the agent revises until accepted.

  2. SQL Synthesis (Stage 2): Using Stage 1 specs, the agent synthesizes SQL queries, executes them locally, and feeds error traces + execution outputs back through the Council's evaluate-and-refine loop until a correct, executable query is produced.

  3. Jinja2 Template Generation (Stage 3): The agent produces a Jinja2 template referencing exact SQL output column names. The Council checks SQL↔template alignment; mismatches trigger joint refinement of both artifacts.

The LLM Council is a heterogeneous ensemble (GPT-4o-mini, Claude 4 Sonnet, DeepSeek v3) using majority voting for acceptance and aggregated feedback for the revision prompt. The final offline template (SQL + Jinja2) is schema-bound and reusable across any table with the same schema, avoiding repeated LLM inference at serving time.

Key Contributions

  • Offline template paradigm: First agentic framework to produce reusable, schema-bound (SQL + Jinja2) artifacts for query-focused table summarization, satisfying all four practical desiderata simultaneously.
  • LLM Council: A heterogeneous multi-LLM ensemble with majority voting and aggregated feedback for iterative artifact validation at each pipeline stage.
  • Privacy-by-design: Only table schemas (column names, types) are sent to LLMs; raw values never leave the local environment.
  • Empirical validation: Consistent best or second-best on FeTaQA, QTSumm, and QFMTS across BLEU, ROUGE-L, and METEOR.

Results

  • FeTaQA: FACTS 32.6 BLEU / 58.9 ROUGE-L / 67.7 METEOR vs. best baseline SPaGe 33.8 / 55.7 / 62.3 (wins on ROUGE-L and METEOR).
  • QTSumm: FACTS 21.9 / 45.8 / 51.3 vs. SPaGe 20.9 / 41.3 / 47.7 (wins all three).
  • QFMTS (multi-table): FACTS 46.0 / 70.8 / 73.2 vs. SPaGe 45.7 / 68.3 / 73.4 (wins BLEU and ROUGE-L, ties METEOR).
  • GPT-Only Council variant (homogeneous) still outperforms most baselines, confirming workflow structure drives gains independent of model diversity.
  • Human evaluation (100 QTSumm + 100 QFMTS examples): 97% intent match, 94% SQL execution accuracy, 98% template rendering accuracy, ~3% Council consensus error rate.
  • Average per-sample cost: 9,922 input tokens, 1,045 output tokens; 2.47 spec items, 1.36 SQL refinement rounds, 1.84 template refinement rounds.

Limitations

  • Template reusability is defined under identical schema only; schema drift (renamed/added columns) breaks templates and is explicitly out of scope.
  • SQL execution accuracy (94%) is the binding constraint on factual correctness—Council approval does not guarantee correct SQL outputs (~6% of Council-approved SQL queries produce incorrect values).
  • Evaluated on three English benchmarks; generalization to other languages, domains, or non-relational table structures is untested.
  • The three-model Council incurs higher latency and cost than single-model baselines; token costs are reported only as averages, not tail distributions.
  • Jinja2 template generation assumes SQL outputs have stable, predictable shapes; empty-result handling requires explicit template branching that the agent may omit.

Relevance to Harnesses / Meta-Harnesses

FACTS is a concrete instance of a structured agentic harness: a fixed, multi-stage orchestration loop where each stage produces an artifact, an ensemble validator (the LLM Council) gates progression, and failure routes back into a controlled refinement loop—exactly the evaluate-and-refine skeleton common to meta-harness designs. The LLM Council pattern—heterogeneous models, majority voting, aggregated feedback as the next-stage prompt—is a reusable meta-harness primitive for quality assurance that could be extracted and applied to other artifact-generation pipelines. The offline template concept (generate once, reuse at scale) is an architectural insight transferable to any harness where LLM calls are a bottleneck: pre-generate execution artifacts against schemas/interfaces, then serve deterministically. The paper also provides concrete token and iteration cost accounting per stage, useful for calibrating harness budget controls.