Data Analysis in the Wild: Benchmarking Large Language Models Against Real-World Data Complexities

🕒 Published (v1): 2026-07-07 16:43 UTC · Source: Arxiv · link

Why this paper was selected

Real-world multi-tabular benchmark exposes evaluation gaps for data analysis agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DataGovBench is a benchmark derived from governmental open-data portals that evaluates LLMs on two tasks: Table QA (decomposable questions with text or visualization answers over large multi-tabular datasets) and Table Insight (proactive expert-level finding generation without an explicit query). Even top-performing closed-source models equipped with agentic scaffolding score below 0.40 on Table QA (Whole) and below 0.50 on Table Insight (Summary), exposing large gaps between current LLM capability and real-world data analytics.

Problem

Existing Table QA benchmarks (WikiTableQuestions, Spider, TableBench) use small, single-table corpora focused on direct fact retrieval; they omit large-scale multi-tabular schemas, metadata, external knowledge files, and decomposable multi-step questions. Insight-generation benchmarks (InsightBench, MT-RAIG) address discovery but still lack large tables, metadata, and external knowledge integration simultaneously. No prior benchmark combines all these real-world data complexities with both QA and open-ended insight tasks.

Method

Data sourcing. 53 English open-data portals (Data.gov, Data.gov.uk, Open.canada, etc.) were systematically filtered to retain datasets with ≥5,000 rows and descriptive metadata, yielding 178 datasets (avg 210K rows, 18 columns; max 11.9M rows, 213 columns). Over 36% of datasets are multi-tabular and 57% include external knowledge in heterogeneous formats (PDF, XLSX, JSON, etc.).

Feature type-specific table serialization. Because tables can exceed LLM context limits, each column is summarized by data type (e.g., unique category set for categoricals, min/max/distribution statistics for numerics) rather than listing raw rows. This compact representation is fed to all downstream LLM calls.

Table QA annotation (4-stage pipeline). 1. Question generation: eight question types defined; ensemble of GPT-4o, GPT-4o-mini, Gemini 2.0 Flash, Gemini 1.5 Pro generates candidates using serialized tables + metadata + external knowledge. 2. Question scoring: each of the four LLMs acts as a judge scoring candidates on relevance, insight potential, complexity, and clarity (max score 20); top-10 per dataset advance. 3. Answer generation: four LLMs generate Python code; questions where all four agree are dropped (only 6%) as insufficiently complex. 4. Human verification: annotators using a custom GUI refine questions, debug code, and validate answers; a second independent reviewer group performs final QA. Yields 211 curated QA pairs.

Table Insight annotation. Six datasets are paired with official domain-expert reports. Insights are extracted as bullet points directly or via NotebookLM prompting on free-text sections, then manually verified and converted to declarative sentences + a summary via Gemini 2.5 Flash.

Answer Agent (inference harness). Three modules: (i) serialization of tables, (ii) coding module that generates Python, with up to 3 self-correction iterations on execution errors, and (iii) reflection module using a VLM/MLLM to detect semantically incorrect outputs (empty plots, NaN results) and trigger up to 3 additional revision loops.

Insight Agent. Extends AgentPoirot's tree-based exploration into a directed acyclic graph (DAG) structure: generates high-level questions → uses Answer Agent for correct answers → synthesizes insights → seeds follow-up questions by aggregating all prior context → terminates with a global summary.

Evaluation metrics. Table QA: Exact Match for text; MLLM-as-judge (majority vote of 4 MLLMs comparing rendered images + source code) for visualizations; Whole (all sub-questions correct) and Individual (per-sub-question) accuracy. Table Insight: LLaMA-3-Eval protocol (replaced with GPT-4o as evaluator) at Summary-level and Insight-level granularity.

Key Contributions

• DataGovBench: first benchmark combining large-scale, multi-tabular, metadata-rich, and externally-supplemented datasets for both decomposable Table QA and open-ended Table Insight.
• Feature type-specific table serialization enabling LLMs to process tables with millions of rows within context limits.
• Comprehensive evaluation of 10 closed- and open-source LLMs with and without agentic scaffolding, establishing clear baselines.
• Qualitative failure-mode analysis identifying two missing capabilities: narrative-level reasoning over tabular data and accurate fact retrieval from complex tables.

Results

• Table QA (Whole / Individual with Answer Agent): Gemini 2.5 Flash 0.393 / 0.502; Claude Sonnet 4.6 0.389 / 0.534; GPT-5.1 0.348 / 0.489; GPT-4o 0.270 / 0.391.
• Table QA (Whole / Individual without Answer Agent): Claude Sonnet 4.6 0.337 / 0.478; GPT-5.1 0.289 / 0.411; Gemini 2.5 Flash 0.310 / 0.401.
• Agentic support yields ~27% relative improvement in Whole accuracy for Gemini 2.5 Flash.
• Top open-source model (Qwen3-30B) achieves 0.199 / 0.309 Whole/Individual with agent, lagging significantly behind closed-source.
• TableGPT2-7B (specialized table model) scores below 0.1 even with Answer Agent.
• Table Insight (AgentPoirot / Insight Agent, Insight-level / Summary-level): Claude Sonnet 4.6 achieves 0.338 / 0.397 (AgentPoirot) and 0.323 / 0.453 (Insight Agent); GPT-5.1 0.295 / 0.355 and 0.311 / 0.401; Gemini 2.5 Flash 0.283 / 0.359 and 0.315 / 0.405.
• All absolute scores remain below 0.40 (Insight-level) and 0.50 (Summary-level), indicating the benchmark is unsaturated.
• Specialized table agents (InfiAgent-DABench, tablegpt-agent) achieved near-zero performance and were excluded.

Limitations

• Table Insight subset is small (6 datasets), limiting statistical power for the insight task.
• Insight ground truth for free-text reports relies on LLM-assisted extraction (NotebookLM), which may introduce extraction noise despite manual verification.
• Error analysis (partial in the provided text) covers only one model configuration (Gemini 2.5 Flash + Answer Agent), reducing generalizability of failure-mode conclusions.
• Benchmark covers only English-language open data portals; domain coverage is governmental, potentially skewing toward administrative/regulatory data patterns.
• The 6% question-removal criterion (unanimous model agreement = low complexity) is heuristic and may discard valid hard questions where all models happen to agree correctly.

Relevance to Harnesses / Meta-Harnesses

DataGovBench directly instantiates a multi-module agentic harness (Answer Agent + Insight Agent) as both the evaluation vehicle and a contribution in its own right, making it a concrete design-space example for harness researchers. The Answer Agent's architecture — serialization → code generation → self-correction loop → reflection loop with VLM validation — is a canonical harness pattern: modular, retry-bounded, and modality-aware. The Insight Agent's shift from tree-structured to DAG-structured exploration (aggregating all prior context rather than a single parent node) is a meta-harness design decision with measurable downstream effects on score. The benchmark's finding that agentic scaffolding provides ~27% relative improvement over raw LLMs, yet still leaves absolute performance low, directly quantifies the headroom remaining for harness design improvements.