Skip to content

InnoGym: Benchmarking the Innovation Potential of AI Agents

🕒 Published (v1): 2025-12-01 16:03 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

InnoGym is the first benchmark targeting the innovation potential of AI agents, measuring both performance gain over known baselines and methodological novelty of submitted solutions. Experiments across 10 tasks show that current agents consistently fail to surpass human state-of-the-art, and that high novelty rarely translates to high performance—robustness is the primary bottleneck.

Problem

Existing benchmarks (MLEBench, SWE-Bench, MLE-Bench, etc.) judge agents solely on answer correctness or leaderboard score, conflating solution value with methodological originality. Two agents can reach identical scores via fundamentally different approaches, yet no benchmark distinguishes this. There is no principled framework for measuring how an agent solves a problem, only whether it does.

Method

The paper formalizes each task as a quadruple T = (P, S, V, D): problem instance, solution space, performance measure V(s) = C(s)·R(s) (feasibility × quality), and a pairwise distance function D. Two metrics are derived: - Performance Gain G(s): improvement over the best-known human/literature baseline V_known. - Novelty N(s)*: minimum distance to any known solution in S_known, computed only for feasible submissions; instantiated via an Agent-as-judge pipeline (Codex extracts strategy profiles; GPT-5 scores methodological dissimilarity on six rubric dimensions, 0–4 each, rescaled to [0,100]).

iBench contains 18 Improvable Tasks (no known optimum, clear room for improvement) curated from NeurIPS Competitions, KDD Cup, ROADEF, GMCM, MLArchSys, and classical NP-hard problems via two-stage filtering (197→72→18): resource/affordability checks then evaluator quality/domain-balance checks. Each task is standardized with Markdown specs, containerized environments, validators, normalized evaluators (Pearson ≥ 0.9, Kendall-τ ≥ 0.8 against original leaderboards), and partitioned dev/eval splits.

iGym is a unified SDK providing asynchronous tool dispatch, checkpoint/resume for long-horizon runs, native concurrency, and stateful/stateless tool abstractions, hosting all agent evaluations.

Agents evaluated: MLAB, CodeAct, AIDE, all running DeepSeek-v3.1 (main); GPT-5 and Gemini-2.5-Pro ablated on CirclePacking. 12-hour wall-clock budget per run, best of 3 valid submissions.

Key Contributions

  • Formal framework defining innovation via (G, N) in the (Performance Gain, Novelty) space, with a taxonomy: breakthrough (high G + high N), performance (high G, low N), conceptual (G≈0, high N).
  • iBench: 18 standardized Improvable Tasks spanning ML, OR, systems, science, and math—first benchmark to jointly evaluate performance and novelty.
  • iGym: unified agent execution environment with recovery, concurrency, and consistent tooling across agent frameworks.
  • Complex-plane representation mapping G (magnitude) and normalized N (angle) to visualize solution development trajectories.
  • Empirical finding that robustness, not idea generation, is the primary bottleneck for current agents.

Results

  • No agent surpassed the human state-of-the-art on any of the 10 evaluated tasks.
  • MLAB achieved average Gain = –24.32, Ratio = –0.45, Novelty = 56.55 (best performance).
  • CodeAct: Gain = –41.58, Ratio = –0.69, Novelty = 54.86.
  • AIDE: Gain = –42.68, Ratio = –0.64, Novelty = 46.67.
  • Several tasks (CDML, PTTALC for MLAB; BEETL-MI, CDML, PTTALC, RCIC for CodeAct) produced zero valid submissions across all runs.
  • On CirclePacking with AIDE + Gemini-2.5-Pro: score 2.49, vs. AlphaEvolve baseline of 2.65; GPT-5 reaches 2.44, DeepSeek-v3.1 reaches 2.40.
  • Temperature sweep: performance gain peaks at low temperature; novelty peaks at high temperature; sweet spot at 0.5–0.75.
  • Extended runs show G increases monotonically over time while N decreases, consistent with convergence to local optima.

Limitations

  • Main experiments cover only 10 of 18 tasks due to compute/engineering constraints; results may not generalize to the excluded 8.
  • Novelty metric relies on an LLM judge (GPT-5 via Codex extraction), introducing model-dependent bias and potential inconsistency.
  • Only three agent frameworks tested; results may not generalize to other architectures.
  • 3 runs per configuration is a small sample for stochastic agents; best-of-3 reporting may overestimate performance.
  • Benchmark restricted to Improvable Tasks; Exploratory and Solved problem types are excluded from evaluation.
  • All tasks are from public competitions (2018–2024), which may overlap with LLM training data, inflating or deflating certain scores unpredictably.

Relevance to Agentic AI / LLM Agents

InnoGym directly probes a blind spot of current agent evaluation: whether agents generate genuinely new methods or merely recombine known ones at high performance. The finding that novelty is abundant but robustness is scarce is a concrete, quantified diagnosis of where agentic systems fall short on long-horizon, open-ended engineering tasks. The iGym environment—with checkpoint/resume and native concurrency—is directly reusable infrastructure for future agent research. The (G, N) framework also offers a template for extending existing ML-engineering benchmarks (MLE-Bench, DSBench) to measure methodological originality, which is critical as agent evaluation moves toward scientific discovery and autonomous R&D.