Skip to content

What Does It Take to Be a Good AI Research Agent? Studying the Role of Ideation Diversity

🕒 Published (v1): 2025-11-19 16:32 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper studies why some AI research agents outperform others on MLE-bench by identifying ideation diversity—the variety of ML architectures an agent proposes during its ideation phase—as a key bottleneck. Through analysis of 11,000 agent trajectories and a controlled prompt-ablation experiment, the authors show that higher ideation diversity causally improves agent performance across multiple metrics.

Problem

AI research agents vary widely in performance, but the factors driving success or failure are poorly understood due to long multi-step trajectories, heuristic search algorithms, and the computational cost of collecting sufficient samples for meaningful ablation. Specifically, it was unknown whether the diversity of ideas generated at the ideation phase affects downstream task performance.

Method

Trajectory analysis: 11,000 trajectories from 6 LLM backbones (o3, GPT-OSS 20B/120B, Llama Maverick, Devstral, CWM) × 2–3 agentic scaffolds (AIDE/Greedy, AIRA-Greedy, AIRA-MCTS) × 75 MLE-bench tasks × 10–20 seeds (≈1.2M scaffold nodes, 264,000 GPU-hours total).

Diversity measurement: Extract the distribution of ML model architectures from an agent's first 5 Draft nodes (ideation phase), then compute Shannon entropy \(H = -\sum_i p_i \log_2 p_i\) over that distribution. A complementary metric, tree-level diversity, counts the mean number of distinct architectures in those 5 nodes.

Controlled experiment: On MLE-bench Lite (22 tasks, 10 seeds, DeepSeek R1 backbone), compare baseline agents—which use sibling memory, prompt-adaptive complexity, and explicit diversity instructions—against ablated agents where complexity cues and diversity instructions are replaced with instructions to generate similar ideas. Only prompt changes are made; implementation quality prompts are held constant.

Key Contributions

  • Proposal of Shannon entropy over Draft-node architecture distributions as a quantitative ideation diversity metric.
  • First large-scale trajectory bank for AI research agents: 11,000 trajectories across 6 LLMs and 3 scaffolds.
  • Observational finding: diversity (entropy) correlates with MLE-bench medal rate at Pearson \(r = 0.57\) (\(p = 4.65 \times 10^{-14}\)), and even more strongly with normalized score (\(r = 0.72\)) and percentile (\(r = 0.66\)).
  • Causal confirmation via prompt ablation: reducing ideation diversity decreases medal rate by 6.9 pp (AIRA-Greedy) and 8.4 pp (AIRA-MCTS).
  • Demonstration that findings are robust across five alternative metrics: valid submission rate, average normalized score, percentile, medal rate, and ELO.

Results

  • Baseline medal rates: AIRAGreedy-R1: 47.0%, AIRA-MCTS-R1: 45.5%.
  • Low-diversity ablations: AIRAGreedy-R1: 38.6% (−8.4 pp), AIRA-MCTS-R1: 38.6% (−6.9 pp).
  • High-performing agents (o3, GPT-OSS) average 3.5 distinct architectures in 5 Draft nodes vs. 2.8 for weaker open-source models (Llama Maverick, Devstral, CWM).
  • Ablation reduces tasks where ≤2 distinct architectures appear from 40% (baseline) to 70% (low-diversity), confirming the prompt manipulation actually shifts diversity.
  • Correlation holds across all 5 evaluation metrics; normalized-score and percentile correlations are higher than medal-rate correlation.

Limitations

  • Diversity is measured only over the model/architecture dimension of the initial Draft nodes; diversity in data preprocessing, feature engineering, and hyperparameter space is ignored.
  • The controlled experiment uses a single LLM backbone (DeepSeek R1) on 22 tasks; generalization to other models or full 75-task benchmark is not directly shown.
  • Diversity is controlled only via prompt; temperature-based control is relegated to the Appendix and may conflate diversity with output stochasticity.
  • Observational correlation confounds ideation diversity with model capability—stronger LLMs are both more diverse and better at implementation, making causal isolation imperfect despite the prompt ablation.
  • The study covers ML engineering competitions (Kaggle/MLE-bench) and may not generalize to open-ended scientific discovery tasks.

Relevance to Harnesses / Meta-Harnesses

This paper directly characterizes the design of agentic scaffolds (meta-harnesses) as a primary lever over agent behavior, showing that prompt engineering choices embedded in the outer loop—sibling memory, prompt-adaptive complexity scheduling, diversity instructions—causally shape what the inner LLM proposes. The scaffold-level mechanisms studied (AIDE, AIRA-Greedy, AIRA-MCTS) are canonical examples of meta-harnesses that orchestrate draft/debug/improve operators, and the paper provides quantitative evidence that harness design choices propagate through to task-level performance. For researchers building or evaluating meta-harnesses, this establishes ideation diversity as a measurable, optimizable objective that should be surfaced in harness diagnostics alongside raw success rate.