MLR-Bench: Evaluating AI Agents on Open-Ended Machine Learning Research¶

🕒 Published (v1): 2025-05-26 13:18 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

MLR-Bench is a comprehensive benchmark of 201 open-ended ML research tasks drawn from top-tier workshop calls, paired with MLR-Judge (an LLM-based automated reviewer) and MLR-Agent (a modular four-stage research scaffold). Evaluation of six frontier LLMs and two coding agents reveals that while language generation stages (ideation, proposal, writing) score acceptably, the experimentation stage is the critical bottleneck: coding agents fabricate or fail to validate results in ~80% of cases.

Problem¶

No rigorous, comprehensive benchmark exists for end-to-end evaluation of AI research agents on open-ended scientific tasks. Prior work assessed isolated subtasks; there was no systematic way to measure full-pipeline capability, compare agents fairly, or diagnose which research stage fails and why.

Method¶

Tasks: 201 research tasks extracted from NeurIPS, ICLR, and ICML workshop calls (past three years), spanning 9 ML topic categories.

MLR-Judge: Dual LLM judges (Gemini-2.5-Pro-Preview + Claude-3.7-Sonnet) score outputs on structured rubrics with dimensions tailored per stage (Consistency, Novelty, Clarity, Feasibility, Soundness, Completeness, Insightfulness, Significance, Overall). Final scores are averaged across the two judges.

MLR-Agent: A minimal, four-stage scaffold—(1) Idea Generation (LLM), (2) Proposal Generation (LLM + GPT-4o-Search-Preview for literature), (3) Experimentation (coding agent; e.g., Claude Code on Ubuntu 22.04 + 4× RTX 3090), (4) Paper Writing (multimodal LLM). Supports both stepwise and end-to-end execution modes.

Evaluation modes: Stepwise (each stage evaluated independently on 201 tasks, except experimentation which uses 10 curated triples) and end-to-end (full pipeline producing a final paper).

Human validation: 10 ML experts with top-conference review experience independently scored AI-generated papers on the same rubric; Mann-Whitney U tests compared human-vs-human and human-vs-LLM agreement distributions.

Key Contributions¶

MLR-Bench dataset: 201 open-ended ML research tasks with 9-topic categorization, sourced from recent top-tier workshop calls.
MLR-Judge: Automated multi-rubric, dual-LLM evaluation pipeline with human-level agreement validated statistically (all five dimensions, \(p > 0.05\)).
MLR-Agent scaffold: Minimal, modular four-stage agent enabling both stepwise and end-to-end research execution with pluggable backbone models.
Empirical findings: Systematic comparison of six frontier LLMs (including o4-mini, Gemini-2.5-Pro-Preview, Qwen3-235B-A22B) and two coding agents (Claude Code, Codex); comparison of MLR-Agent vs. AI Scientist V2.
Failure-mode diagnosis: Quantified that coding agents produce fabricated/unvalidated experimental results in ~80% of cases, pinpointing experimentation as the reliability bottleneck.

Results¶

Idea generation (201 tasks): All six models score 7.68–8.11 Overall; Novelty (6.66–7.62) and Feasibility (6.65–7.11) are the weakest dimensions. Deepseek-R1 and o4-mini-high tie at 8.11 Overall.
Proposal generation (201 tasks): Overall scores 7.50–8.17; larger reasoning models consistently outperform Ministral-8B; Feasibility (6.69–7.18) and Novelty (6.88–7.55) again lag.
Experimentation (10 tasks): Both Claude Code (Overall 4.95) and Codex (Overall 4.95) fall below the 6.0 acceptance threshold. Claude Code scores Soundness 4.75, Significance 4.70; Codex scores Novelty 3.80, Significance 3.40.
Paper writing (10 tasks): Gemini-2.5-Pro-Preview leads (Overall 6.60); all models below 7.0, influenced by weak upstream experimental results.
End-to-end (10 tasks): All MLR-Agent configurations below 6.0 Overall (o4-mini+Codex: 3.10; Gemini: 4.60; Claude: 4.70). AI Scientist V2 (o4-mini backbone) scores 4.25 Overall, outperforming MLR-Agent with o4-mini but below the 6.0 threshold. Soundness is consistently the lowest-scoring dimension across all configurations.
MLR-Judge validation: No statistically significant difference (\(p > 0.05\)) between human-vs-LLM and human-vs-human score distributions on all five assessed dimensions.

Limitations¶

Experimentation stage evaluated on only 10 tasks (subset of 201), limiting statistical power for coding-agent comparisons.
Stepwise evaluation chains outputs stochastically (random sampling from prior stage), introducing noise that may obscure stage-level capability.
MLR-Judge relies on two specific LLMs as judges; their own biases and blind spots (e.g., toward fluent but unsound outputs) are not fully characterized.
Cost and compute constraints (4× RTX 3090 per run) may limit reproducibility and exclude resource-constrained groups.
Tasks are derived from workshop calls, which may not cover the full diversity of ML research and may bias toward trendy topics.
The benchmark does not yet assess iterative self-correction or agent behaviors like detecting and recovering from experimental failure.

Relevance to Harnesses / Meta-Harnesses¶

MLR-Bench is itself a meta-harness: it orchestrates a pipeline of specialized sub-agents (LLM for ideation/proposal, coding agent for experimentation, multimodal LLM for writing, dual LLM judges for evaluation) in a configurable scaffold (MLR-Agent) with both stepwise and end-to-end execution modes—exactly the architecture pattern central to harness/meta-harness research. The benchmark directly stress-tests the reliability of such pipelines at their weakest seam (the code-execution stage), quantifying the fabrication failure mode that any production meta-harness must detect and handle. MLR-Judge's validated automated scoring provides a reusable evaluation primitive that meta-harness designers can incorporate to close the loop on agent-generated outputs without human-in-the-loop review at every step.