Skip to content

StatABench: Dataset and Framework for Evaluating Statistical Analysis Capabilities of LLMs

🕒 Published (v1): 2026-06-22 07:57 UTC · Source: Arxiv · link

Why this paper was selected

StatABench evaluation framework for LLM tool-use in statistical analysis; benchmark harness design reusable across domains

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

StatABench is a two-track benchmark for evaluating LLMs on statistical analysis tasks, combining 404 closed-form questions (Stat-Closed) with 30 open-ended real-world modeling problems (Stat-Open). It introduces a dynamic LLM-as-Judge evaluation protocol and shows that even GPT-5.1 reaches only 68.6% on Stat-Closed, exposing a substantial gap between current LLMs and reliable statistical reasoning. The benchmark is uniquely distinguished by requiring tool use through a LangChain MCP framework alongside domain coverage.

Problem

Existing statistical benchmarks (StatQA, QRData, StatLLM, StatEval) are narrow in scope, limited to static evaluation formats, or decouple conceptual reasoning from tool use — failing to capture the dual requirement of real-world statistical analysis, where practitioners must both reason about methods and operationalize them through software (R, Python, SPSS). No prior benchmark combined agentic tool use, open-ended modeling, and dynamic evaluation in a single statistical-domain harness.

Method

Stat-Closed: 404 questions across 18 statistical topics in four formats (multiple-choice, fill-in-the-blank, decision-making, practical application). A 35-function SAToolKit is exposed via LangChain MCP, allowing models to invoke statistical procedures. Practical application questions are triplets of (question, dataset, ground-truth function parameters) generated via Gemini 2.5 Pro through dataset-driven adaptation or synthetic generation, then manually reviewed. A decontamination pipeline flags questions solvable by Qwen2.5-72B or Llama-3.1-70B and perturbs them until at least one model fails.

Stat-Open: 30 end-to-end statistical modeling tasks drawn from MCM/ICM, CUMCM, and MAS competitions. Agents (MathModelAgent, LLM-MM-Agent) perform full pipelines: data cleaning → EDA → method selection → code execution → report generation. Reports are scored by Gemini 3 Pro as LLM-as-Judge on four dimensions: Structural Coherency, Methodological Quality, Practicality and Scientific Validity, and Result Interpretation/Bias Analysis — each 0–1, equally weighted.

Baselines evaluated: Qwen3-8B, Qwen2.5 (7B/32B/72B), Llama-3.1-8B, DeepSeek-V3, GPT-4o-mini, GPT-5.1, Claude 4.5. Four agent frameworks on Stat-Closed practical tasks: Qwen Agent, CrewAI, AutoGen, Smolagents.

Key Contributions

  • SAToolKit: 35 callable statistical functions covering all 18 topics, exposed uniformly via LangChain MCP for controlled tool-use evaluation.
  • Decontamination pipeline: Modify-and-verify procedure that perturbs leaked questions until model failure is confirmed, reducing memorization bias.
  • Dual-track benchmark: Stat-Closed (ground-truth matching) + Stat-Open (LLM-as-Judge on structured reports) in one framework.
  • Validated LLM-as-Judge: Gemini 3 Pro selected as judge by highest Fleiss' \(\kappa\) (\(\bar{\kappa} = 0.51\)) vs. GPT-5 (\(0.25\)) and Grok-4 (\(0.19\)) on human-annotated subsets.
  • Framework-vs-model decomposition: Shows that agentic orchestration raises absolute scores but does not eliminate cross-model statistical capability gaps.

Results

  • GPT-5.1 achieves 68.6% overall on Stat-Closed; best open-source (Qwen2.5-72B) reaches 60.6%; Llama-3.1-8B reaches only 25.5%.
  • Friedman test across 18 categories: \(p = 2.87 \times 10^{-9}\), confirming differences are not random.
  • All models exhibit a tool-integration tax: consistent accuracy drop from fundamental to practical tasks; especially severe for smaller models (Llama-3.1-8B, GPT-4o-mini).
  • Agent frameworks improve practical task accuracy substantially: DeepSeek-V3 + CrewAI reaches 85.35% on Stat-Closed practical questions; most model–framework combinations cluster at 50–60%.
  • Cross-model failure analysis: 74% of cases where DeepSeek-V3 fails but Qwen2.5-72B succeeds are statistical errors (wrong tool selection, misread p-values), not engineering errors.
  • Stat-Open: MathModelAgent (DeepSeek-V3 backbone) scores 61.86/100 average; LLM-MM-Agent scores 54.29/100.
  • MathModelAgent vs. LLM-MM-Agent trade-off: template-based agent scores higher on Structural Coherency (71.50 vs. 34.00) and Practical Science (91.00 vs. 69.00), but far lower on Innovativeness (14.50 vs. 65.50).

Limitations

  • Stat-Open contains only 30 tasks, limiting statistical power for fine-grained framework comparisons.
  • Innovativeness dimension has low inter-annotator agreement (\(\kappa \approx 0.2\)–\(0.4\)), introducing subjectivity in the judge protocol.
  • Only two agent frameworks evaluated on Stat-Open (MathModelAgent, LLM-MM-Agent); broader coverage would strengthen conclusions about orchestration effects.
  • SAToolKit covers 35 functions; real statistical workflows often involve bespoke pipelines, package interactions, and iterative debugging not captured here.
  • The decontamination pipeline is heuristic: models that happen to fail a contaminated question may still have seen it in training.

Relevance to Harnesses / Meta-Harnesses

StatABench directly instantiates a meta-harness pattern: it wraps multiple agentic frameworks (LangChain MCP, CrewAI, AutoGen, Qwen Agent, Smolagents) under a unified evaluation harness, using LLM-as-Judge as a dynamic scoring layer on top of heterogeneous agent outputs. The SAToolKit-as-MCP-server design is a reusable harness component — a controlled tool environment that any agent framework plugs into, enabling apples-to-apples comparison of orchestration strategies. The key finding for harness researchers is that framework choice rescales but does not erase underlying model capability gaps, meaning evaluation harnesses must separately account for orchestration quality and base model reasoning. The validated multi-dimensional LLM-as-Judge protocol (with \(\kappa\) selection by judge model) is a transferable harness design pattern for evaluating open-ended agentic outputs in any domain.