Skip to content

AstaBench: Rigorous Benchmarking of AI Agents with a Scientific Research Suite

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Allen AI; rigorous benchmark for AI co-scientist agents on scientific tasks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AstaBench is a rigorous benchmark suite for evaluating AI agents on holistic scientific research tasks, comprising 2,400+ problems across 11 benchmarks spanning literature understanding, code execution, data analysis, and end-to-end discovery. It ships with production-grade reproducible search tools, standardized agent interfaces, and cost-aware leaderboard infrastructure. Evaluation of 57 agents across 22 agent classes reveals that the best agent (Asta v0) achieves ~53% overall, demonstrating that science research assistance remains far from solved.

Problem

Existing agent benchmarks fail to enable rigorous, controlled comparison of AI agents along five axes: (1) lack of standardized, production-grade tool environments that decouple information-access advantage from agent capability; (2) failure to account for confounders such as inference cost and tool access; (3) benchmark task formats that are coupled to specific agent frameworks, hampering general agent integration; (4) absence of product-usage-grounded tasks, risking disconnect from real-world value; and (5) insufficient diversity of agent baselines, making claimed improvements hard to contextualize.

Method

AstaBench packages 11 benchmarks into four task categories—Literature Understanding, Code & Execution, Data Analysis, and End-to-End Discovery—covering tasks such as paper retrieval (PaperFindingBench), long-form scientific QA (ScholarQA-CS2), literature table synthesis (ArxivDIGESTables-Clean), ML experiment replication (SUPER-Expert), data hypothesis discovery (DiscoveryBench), and full pipeline research projects (E2E-Bench/E2E-Bench-Hard). All benchmarks share a uniform task interface (Inspect-compatible) and are paired with the Asta Environment: production-grade scientific corpus tools (snippet_search, paper retrieval, citation graph, author search) exposed via MCP, plus a stateful Jupyter computational notebook sandbox. The corpus tools accept date cutoffs to prevent contamination from post-creation papers. Cost accounting is handled by the agent-eval toolkit, which maps logged model usages to a frozen LitLLM price snapshot, ensuring time-invariant fair cost comparison and cache discount handling. The leaderboard further categorizes submissions by agent openness (open-source vs. closed) and tool conformance (standard vs. custom). Nine Asta agent classes (specialized per task category, e.g., Asta Paper Finder, Asta CodeScientist, Asta v0) plus numerous baselines (ReAct, direct-answer, etc.) are distributed as open-source agent-baselines.

Key Contributions

  • AstaBench suite: 2,400+ problems across 11 benchmarks and 4 scientific task categories, including tasks grounded in real user requests from the deployed Asta product.
  • Asta Environment: First reproducible, production-grade scientific literature search toolset (8 MCP tools + stateful Jupyter notebook) that decouples information access from core agent capability.
  • agent-eval toolkit: Time-invariant cost accounting over model usage logs (frozen LitLLM price snapshot), benchmark suite layer over Inspect, and a public leaderboard with confounder-aware reporting.
  • agent-baselines suite: Nine science-optimized Asta agent classes plus baselines, all open-source with Inspect-compatible interfaces—the largest standardized multi-class agent suite to date.
  • Systematic large-scale evaluation: 57 agents across 22 agent classes, the broadest evaluation of science-research agents published to date.
  • Benchmarking principles (Appendix A) formalizing criteria for rigorous agent evaluation.

Results

  • Best overall agent: Asta v0 (fully custom tooling) scores ~53.0% across all benchmarks.
  • Best general-purpose agent: ReAct with gpt-5 scores ~44.0%—roughly 9 percentage points below Asta v0.
  • gpt-5 vs. o3: gpt-5 provides only a modest improvement over o3 on most benchmarks; it actually hurts performance for several specialized agents.
  • Open vs. closed weights: Agents powered by closed-weight LLMs substantially outperform open-weight-based agents across the suite.
  • Commercial science agents: Best API-based commercial agent (FutureHouse Falcon) cannot attempt the full range of AstaBench tasks.
  • Science research assistance remains far from solved at the suite level despite meaningful progress on individual subtasks.

Limitations

  • Evaluation was conducted over several months, introducing temporal variation in underlying LLM capabilities and API costs across runs.
  • CORE-Bench-Hard− omits GPU-requiring tasks from the original CORE-Bench-Hard, reducing coverage of compute-intensive reproduction.
  • The agent-eval CLI leaderboard requires authentication currently unavailable to the public.
  • Cost accounting uses a frozen LitLLM snapshot that may not perfectly reflect actual invoiced costs; latency-based discounts (batching, service tier) are excluded.
  • Task coverage is weighted toward CS/NLP domains (ScholarQA-CS2, SUPER-Expert, E2E-Bench are all CS-only); biological and other scientific domains are represented mainly through LitQA2 and partially through DiscoveryBench.
  • The benchmark tests agentic scaffolding but still relies on LLM judges for several rubric-based tasks, introducing judge-model variance.

Relevance to Agentic AI / LLM Agents

AstaBench directly addresses a critical gap in the agentic AI ecosystem: the absence of a controlled, cost-aware, reproducible benchmark for evaluating LLM agents on complex, multi-step, tool-using tasks. By decoupling information access (via standardized MCP tools) from agent intelligence, it enables meaningful comparisons of agent architectures rather than their retrieval pipelines—a methodological advance for the field. The inclusion of end-to-end discovery tasks (idea → experiment → report) and cost-aware Pareto frontier reporting pushes agent evaluation toward real-world utility metrics rather than narrow accuracy proxies. For researchers building or comparing agentic systems, the open-source agent-baselines suite and confounder-controlled leaderboard provide the most comprehensive standardized testbed currently available for scientific research agents.