Skip to content

AgenticDataBench: A Comprehensive Benchmark for Data Agents

🕒 Published (v1): 2026-07-02 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

Comprehensive benchmark for data agents across heterogeneous tasks — strong DB group (Tsinghua)

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AgenticDataBench is a benchmark for LLM-based data agents covering 433 hierarchically organized data science skills across 15 domains, including 5 real-world B2B fintech tasks. It introduces a hierarchical skill extraction pipeline (embedding clustering + LLM refinement) and skill-coverage-driven task generation to ensure both realism and completeness. Evaluations yield fine-grained skill-level diagnostics rather than coarse aggregate scores.

Problem

Existing data agent benchmarks (DSBench, BLADE, DA-Code, DataSciBench, ScienceAgentBench, KramaBench) cover at most 11 task domains and 281 skills, rely on manually curated or competition-derived tasks, and report only aggregate scores that obscure where agents fail. No benchmark captures the complexity of real industrial data science workflows (large noisy datasets, implicit domain knowledge, diverse file formats) while providing fine-grained, skill-annotated ground truth.

Method

Skill extraction. 6,510 Stack Overflow solutions are decomposed by LLM into 29,602 stepwise skill descriptions. These are embedded with Qwen3-Embedding + UMAP, soft-clustered via GMM, and then each cluster is LLM-refined (split into coherent higher-level skill abstractions). Synonymous skills are merged using DBSCAN with a strict cosine-distance threshold. The cluster-and-refine procedure is applied recursively until the skill count falls below a threshold, yielding a 7-category, 433-node hierarchical skill tree.

Task selection (real-world). 102 tasks from Ant Group are selected by greedily maximizing the number of newly covered skills at each step—a greedy approximation to the NP-hard maximum coverage problem.

Task generation (public datasets). A skill graph weighted by frequency of co-occurrence in solutions is constructed. Skill compositions are sampled with penalties for already-covered skills. Conditioned on a structured dataset profile and a skill-based workflow, an LLM generates task descriptions. Each generated task is expert-annotated with solution code and an evaluation method.

Evaluation. Each benchmark instance is a quintuple \((\delta_t, D, y, S, \text{eval})\). The evaluation function \(\text{eval}: \hat{y} \to [0,1]\) is task-specific (e.g., normalized MSE on monthly metrics for the loan risk task). Results are aggregated per skill, enabling skill-level leaderboards.

Key Contributions

  • A 7-category, 433-skill hierarchical data science skill framework derived data-driven from Stack Overflow at scale.
  • AgenticDataBench: 344 tasks across 15 domains, 27.3 GB data, 18 file formats, avg. 23.5 skills/task and 113.6 lines of ground-truth code/task—larger and more complex than all prior benchmarks.
  • Hierarchical skill extraction algorithm combining GMM-based embedding clustering with LLM cluster refinement and DBSCAN-based synonym merging.
  • Skill-diverse task selection (greedy max-coverage) and skill-coverage-driven task generation (skill graph sampling + structured profiling).
  • Open-sourced testbed with Docker/Bash/Python/database execution environments and skill-level evaluation pipeline.

Results

  • The paper reports a leaderboard including CodeX, Smolagents, and DA-Agent, but specific accuracy numbers are not included in the provided text excerpt.
  • Qualitatively, the paper identifies four key insights from fine-grained empirical analysis; these are referenced but not enumerated in the provided text.
  • AgenticDataBench exceeds all prior benchmarks on every scale dimension: 433 skills vs. 281 (DataSciBench) and 220 (ScienceAgentBench); 113.6 avg. solution lines vs. 85 (DA-Code); 493.4 MB data/task vs. 11.1 MB (DSBench).

Limitations

  • Task generation relies on LLM quality; generated tasks require expert annotation and refinement, limiting scalability of the curation process (>1,560 person-hours reported).
  • Real business tasks are drawn exclusively from one fintech company (Ant Group), potentially introducing domain bias toward financial data patterns.
  • Skill annotations are applied by prompting LLMs, then refined by experts—noisy or missing skill labels remain a risk.
  • The provided text does not include quantitative results, making it impossible to verify the empirical claims about agent performance or the four stated insights.
  • The benchmark is code-execution-centric; tasks requiring interactive data exploration or visualization judgment are harder to auto-evaluate.

Relevance to Agentic AI / LLM Agents

AgenticDataBench directly addresses the evaluation gap for data science agents—a primary application domain for LLM agents involving multi-step planning, iterative code execution, and tool use. The skill-level diagnostic framework enables researchers to pinpoint which agentic capabilities (e.g., error handling, cross-stage reasoning, model deployment) are bottlenecks, going beyond the pass/fail granularity typical of existing agent benchmarks. The inclusion of real industrial tasks with large, noisy datasets and implicit business rules tests agent robustness beyond curated academic settings. The hierarchical skill abstraction methodology is transferable to building fine-grained benchmarks in other agentic domains.