Skip to content

WebDS: An End-to-End Benchmark for Web-based Data Science

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; WebDS: end-to-end benchmark for web-based data science agent tasks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WebDS is an 870-task benchmark spanning 29 real websites that evaluates LLM agents on the full web-based data science pipeline—from autonomous data discovery through analysis to report generation. Current SOTA agents score at most 13.2% on WebDS versus ~90% human accuracy, exposing a massive capability gap that simpler web benchmarks (where the same agents achieve 80%+) entirely miss.

Problem

Existing web-agent benchmarks (WebVoyager, WebArena) test narrow, shallow interactions without data analysis, while data science benchmarks (InfiAgent-DABench, DSBench) assume pre-collected structured datasets. No benchmark evaluates the full pipeline: finding heterogeneous data on live websites, cleaning and analyzing it with tools, and synthesizing an output. This gap hides critical failure modes and makes it impossible to track progress toward practical web-based data science agents.

Method

WebDS defines a web data science task as a composite mapping \(f = f_\alpha \circ f_a \circ f_d\) where \(f_d: W \times Q \to D\) extracts raw data via browser navigation, \(f_a: D \to Y\) transforms it into analytical output (reports, visualizations, predictions), and \(f_\alpha: Y \to A\) optionally performs a downstream action. Tasks were hand-authored by domain experts across 10 domains and 29 websites (government portals, news media, financial data, CDC, etc.) with 7 orthogonal attribute labels (QA, action-based, single/multi-hop, structured/unstructured, tool-use, web-nav, multi-website). Evaluation combines binary automated scoring against ground-truth references and a full-trajectory LLM-as-judge producing a 1–5 score over (observation, action, next-observation) triplets—not just terminal screenshots. A 470-task public validation set and 400-task private held-out test set guard against overfitting.

Key Contributions

  • First end-to-end benchmark covering the full data science workflow: web data acquisition → multi-tool analysis → output generation, across 870 tasks on 29 websites.
  • Dual-track evaluation: WebDS-live (live websites, evolving realism) and WebDS-dockerized (containerized snapshots, full reproducibility)—the most comprehensive dockerized web benchmark to date.
  • Fine-grained three-dimensional evaluation: domain-wise, attribute-wise, and difficulty-tier scoring (easy/medium/hard), enabling diagnostic analysis beyond binary pass/fail.
  • Quantification of the human–agent gap (90% vs. 13.2%) and identification of novel failure modes: poor information grounding, repetitive behavior, and shortcut-taking.

Results

  • Best agent: BrowserUse + GPT-4o achieves 13.2% on WebDS vs. 81.1% on WebVoyager (same agent).
  • AgentOccam + GPT-4o: 4.8% on WebDS vs. 45.7% on WebArena.
  • Human baseline: ~90% (±3%) under identical constraints.
  • Scaling model capacity does not help: GPT-4o performs similarly to GPT-4o-mini and Qwen2.5-72B.
  • Agents perform approximately 2.5Ă— better on easy tasks than medium or hard tasks.
  • LLM judge stability: 96.3% status agreement and 91.3% exact score agreement across 5 repeated runs; per-run average score std = 0.005; 93% agreement with human annotators on 400 task-trajectory pairs.

Limitations

  • WebDS-live tasks are subject to website changes over time, introducing evaluation drift.
  • Task authorship is limited to 8 primary annotators (paper authors), which may introduce coverage biases toward their domain expertise.
  • Action space is intentionally unconstrained, making direct cross-agent comparison on tool use harder to control.
  • Human participants were given up to 30 minutes per task and used live sites at a single point in time; this baseline may not generalize.
  • Private test-set rotation mitigates but does not eliminate leakage risk as the benchmark matures.

Relevance to Agentic AI / LLM Agents

WebDS directly stress-tests long-horizon, multi-tool agents on realistic open-ended workflows—exactly the regime where current agents are claimed to be most capable. The finding that top agents scoring 80%+ on prior benchmarks collapse to 13% here reveals that benchmark saturation has been masking fundamental gaps in grounding, planning persistence, and tool-integrated reasoning. The failure modes identified (information grounding failures, repetitive loops, shortcut-taking) are actionable targets for the agent-design community. The dual live/dockerized evaluation infrastructure also provides a reusable scaffold for future agentic benchmarks that need both reproducibility and ecological validity.