Skip to content

DSBench: How Far Are Data Science Agents from Becoming Data Science Experts?

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DSBench is a realistic benchmark for evaluating data science agents, comprising 466 data analysis tasks (from ModelOff financial competitions) and 74 data modeling tasks (from Kaggle). State-of-the-art LLMs, LVLMs, and agent systems struggle severely: the best agent achieves only 34.12% accuracy on data analysis and a 34.74% Relative Performance Gap (RPG) on data modeling, versus ~64% human performance.

Problem

Existing data science benchmarks (DS-1000, DSEval, etc.) operate under simplified settings—short single-modality instructions, code-completion-only evaluation, fixed execution environments—that do not reflect real-world data science tasks involving long multimodal contexts, large multi-table Excel files, and end-to-end model building. This gap makes it impossible to measure whether current LLMs/LVLMs can function as practical data science agents.

Method

DSBench collects tasks from two competition platforms: ModelOff (financial data analysis, yielding 466 multiple-choice and fill-in-the-blank questions paired with Excel files, tables, and images) and Kaggle (74 ML modeling competitions with training/test splits at 8:2). For data analysis, semantic answer comparison via an LLM evaluator provides task-level and competition-level accuracy. For data modeling, the paper introduces the Relative Performance Gap (RPG): RPG = (1/N) Σ max((pᵢ − bᵢ)/(gᵢ − bᵢ), 0), normalizing each competition's metric against a baseline bᵢ and the human-expert ceiling gᵢ. Evaluated agents include vanilla LLMs/LVLMs (Llama3, GPT-3.5/4/4o, Claude, Gemini, LLaVA) and agent frameworks (AutoGen, Code Interpreter).

Key Contributions

  • DSBench: 540 total tasks with multimodal inputs (text, images, Excel, multi-sheet tables), average task description length 797.9 words—substantially longer and richer than prior benchmarks.
  • Relative Performance Gap (RPG) metric that normalizes heterogeneous evaluation metrics (18 distinct metrics across Kaggle tasks) into a single comparable score.
  • Comprehensive evaluation of representative LLMs, LVLMs, and agent frameworks including GPT-4o, Claude, Gemini, AutoGen, and Code Interpreter, with cost and inference time breakdowns.
  • Empirical findings quantifying the human–agent gap and identifying difficulty factors (input context length, task release year).

Results

  • Best data analysis result: AutoGen + GPT-4o at 34.12% task-level accuracy vs. human ~64.06%.
  • Best data modeling result: AutoGen + GPT-4 at 45.52% RPG; AutoGen + GPT-4o at 34.74% RPG; human baseline ~65.02% RPG.
  • Gemini achieves the highest model-only data analysis accuracy at 31.55%; GPT-4o model-only reaches 28.11%.
  • AutoGen consistently outperforms its underlying vanilla model (e.g., AutoGen+GPT-4o > GPT-4o alone), confirming that agent interaction mechanisms add value.
  • Data analysis accuracy degrades sharply with context length: AutoGen+GPT-4o drops from 74% (<500 tokens input) to 13% (>20k tokens).
  • Data modeling task success is near-independent of input length and task release year; AutoGen+GPT-4 achieves 87.84% task success rate but only 45.52% RPG, indicating file generation ≠ quality modeling.

Limitations

  • Data analysis subset is limited to financial/Excel-centric tasks from ModelOff; domain diversity is narrow.
  • Data modeling uses only 74 Kaggle competitions; test sets are self-split rather than official, which may diverge from true competition difficulty.
  • Human performance estimates are based on small samples (10 competitions for analysis, 22 for modeling) using pre-existing submitted code, not controlled experiments.
  • LLM-based semantic evaluator for fill-in-the-blank answers introduces potential evaluation noise.
  • No LVLM-specific agent framework is evaluated; multimodal image content in tasks may be underutilized by purely text-based agent pipelines.

Relevance to Vision-Language Models

DSBench directly evaluates LVLMs (LLaVA, GPT-4o, Gemini) on tasks that include image modalities alongside tabular and textual data, exposing their limitations in grounded, tool-augmented reasoning rather than perception alone. The benchmark surfaces a critical finding: current LVLMs cannot reliably combine visual understanding with numerical reasoning over large Excel/table structures even when agent scaffolding is provided. For VLM researchers, this establishes a hard, ecologically valid evaluation frontier beyond standard VQA or chart-reading tasks, and motivates work on long-context multimodal grounding and tool-use integration.