HeurekaBench: A Benchmarking Framework for AI Co-scientist

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Maria Brbic (EPFL); realistic end-to-end benchmark for AI co-scientist agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

HeurekaBench is a framework for constructing benchmarks that evaluate LLM-based agents as AI co-scientists on open-ended, data-driven scientific questions. It grounds benchmark construction in peer-reviewed publications and validated code repositories, producing both open-ended and multiple-choice questions. Instantiated in single-cell biology (sc-HeurekaBench: 50 OEQs, 50 MCQs), it reveals that critic modules can improve ill-formed open-source agent responses by up to 22%.

Problem

Existing benchmarks for scientific agents test narrow, instruction-following subtasks (e.g., "how many miRNAs pass FDR correction?") or rely on LLM-generated questions without validation, failing to capture the open-ended, exploratory, multi-step reasoning that defines real co-scientist work. No benchmark required agents to autonomously plan and execute full analysis workflows culminating in data-driven scientific interpretations grounded in experimental datasets.

Method

HeurekaBench constructs \((D, Q, A)\) triplets from peer-reviewed papers paired with open-source code repositories and datasets, via a four-module multi-LLM pipeline:

1. InsightExtractor — generates candidate scientific insights (summary, experimental techniques, verbatim grounding text) from a paper.
2. CodeDescriber — converts repository scripts into natural-language summaries.
3. CodeMatcher — links insights to relevant code files.
4. CodeGenerator — composes a multi-step executable workflow per insight.

Human reviewers run each workflow and validate reproducibility against reported figures/statistics; only reproducible insights are retained. From validated insights, few-shot prompting derives OEQs (exploratory, free-form) and MCQs (with hard distractors). OEQs are filtered to remove questions solvable from pretraining knowledge alone. Evaluation of OEQ responses uses G-Eval with GPT-4o as judge, instructed to decompose both agent and ground-truth responses into atomic facts and score overlap (1–5 scale); additional non-conflicting findings are not penalized.

Key Contributions

• A general semi-automated framework (HeurekaBench) for constructing scientifically grounded, end-to-end co-scientist benchmarks from published papers and code repositories.
• A multi-LLM pipeline with human validation that ensures only reproducible insights enter the benchmark, addressing reliability gaps in prior work.
• sc-HeurekaBench: 50 OEQs and 50 MCQs across 41 validated insights from 13 Nature/Cell single-cell biology papers (2024–2025).
• An atomic-fact decomposition evaluation scheme (G-Eval) that rewards dataset-backed reasoning over literature recall.
• sc-HeurekaBench-ToolUsage (sc-HeurekaBench-TU): 12 OEQs requiring domain-specific bioinformatics tools (SCENIC, CellPhoneDB, etc.).
• Quantitative ablation showing critic modules close the open-source/closed-source performance gap by up to 22% on ill-formed responses.

Results

• InsightExtractor module (FlyBase, 50 pairs): 44/50 strongly related, 2 weakly related, 4 unrelated to expert findings.
• InsightExtractor module (BixBench, 21 papers): 14/21 strongly, 4 weakly, 3 unrelated.
• CodeMatcher module (InsightBench, 215 scripts): 158/215 scripts correctly matched; average 74.6% files retrieved correctly per insight.
• Critic module ablation: adding a critic at the end of the agent loop improves ill-formed responses for open-source LLM-based agents by up to 22%, narrowing the gap with closed-source counterparts.
• Agent comparison (Biomni-A1, CellVoyager, BixBench-Agent) evaluated on sc-HeurekaBench subset (datasets < 750 MB); specific OEQ scores not fully reported in the provided text excerpt.
• Closed-source vs. open-source planners: closed-source models outperform open-source, but multi-turn reasoning narrows the gap.

Limitations

• Framework instantiated only in single-cell RNA-seq biology; generalization to other scientific domains is demonstrated conceptually but not empirically.
• Validation pipeline yields 41 validated insights from 22 candidate papers (roughly 50% retention), discarding insights that require domain-specific tools or additional data not captured by CodeGenerator.
• Benchmark size is small (50 OEQs, 50 MCQs), constrained by the manual review bottleneck.
• GPT-4o as LLM-judge introduces model-specific biases; alignment with human graders is studied but LLM-judge limitations remain.
• Questions restricted to datasets < 750 MB for agent evaluation, potentially excluding harder large-scale analyses.
• Restricting to 2024–2025 Nature/Cell papers partially (not fully) mitigates memorization; contamination cannot be ruled out.

Relevance to Agentic AI / LLM Agents

HeurekaBench directly addresses the evaluation gap for scientific agents operating in the think-act-observe loop over real experimental data, providing the first benchmark that requires agents to autonomously plan and execute multi-step analysis workflows rather than answer single prescribed questions. The atomic-fact evaluation protocol offers a reusable paradigm for scoring free-form agentic outputs against structured ground truth, applicable beyond biology. The ablation of planner, critic, and retriever components quantifies how standard agent design patterns (critic feedback, multi-turn reasoning, retrieval augmentation) affect end-to-end scientific task performance, directly informing architecture choices for LLM agent builders. The framework's semi-automated pipeline for grounding benchmarks in real code repositories is a generalizable methodology for constructing evaluation suites in any data-driven scientific domain.