Skip to content

Paper2Code: Automating Code Generation from Scientific Papers in Machine Learning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Sung Ju Hwang (KAIST); automating code generation from ML papers via agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PaperCoder is a multi-agent LLM framework that transforms ML research papers into executable code repositories through three sequential stages: planning, analysis, and code generation. It addresses the reproducibility gap — only ~19.5% of top-venue ML papers release code — by generating faithful implementations without relying on any pre-existing code artifacts. On both automated and human evaluations, PaperCoder substantially outperforms multi-agent software-engineering baselines (ChatDev, MetaGPT) and naive one-shot generation.

Problem

The majority of ML papers (~80%) do not release code, forcing researchers to manually reverse-engineer implementations from prose — a slow, error-prone process. Prior LLM-based approaches to code generation either target single-file snippets or require existing partial implementations/APIs as scaffolding, leaving the cold-start paper-to-repository problem unsolved.

Method

PaperCoder decomposes the mapping \(M(R) = C\) (paper \(R\) → repository \(C\)) into three stages, each handled by specialized LLM agents:

  1. Planning (\(M_{\text{plan}}(R) \to P = \{o, d, l, g\}\)): Four sequential sub-agents produce (a) an overall implementation plan \(o\), (b) architecture design \(d\) with UML class and sequence diagrams, (c) logic design \(l\) with an ordered file list specifying inter-file execution dependencies, and (d) a config.yaml \(g\) capturing hyperparameters.

  2. Analysis (\(M_{\text{analysis}}(R, P, f_i) \to a_i\) for each file \(f_i\)): Iterates over every file identified in planning and produces a detailed per-file spec covering functional goals, I/O signatures, intra- and inter-file dependencies, and algorithmic constraints.

  3. Coding (\(M_{\text{code}}(R, P, f_i, a_i, \{c_1,\ldots,c_{i-1}\}) \to c_i\)): Generates files sequentially in the execution order determined during logic design, conditioning each file on all prior artifacts (paper, plan, analysis, and already-generated files) to enforce cross-file consistency.

Key Contributions

  • PaperCoder: A three-stage, multi-agent framework for cold-start paper-to-repository code generation with no dependency on pre-existing code artifacts.
  • Paper2CodeBench: A curated benchmark of 90 ML papers (30 each from ICLR, ICML, NeurIPS 2024) with reference-based and reference-free evaluation protocols.
  • Dual evaluation protocol: Reference-based scoring (vs. author-released repos, judged by o3-mini-high with 8-sample averaging) and reference-free scoring (paper-only), with human evaluations from original paper authors to calibrate LLM judges.
  • Demonstration that the structured plan-then-analyze-then-code decomposition is necessary: ablations (Abstract-only, one-shot Paper) consistently underperform the full pipeline.

Results

  • Reference-based (5-point Likert, Paper2CodeBench): PaperCoder 3.68–3.83 vs. ChatDev 2.70–2.97, MetaGPT 2.48–2.95, one-shot Paper 3.08–3.28; statistically significant (\(p \le 0.05\)).
  • Reference-free: PaperCoder 4.73–4.77 vs. Oracle (author repo) 4.80–4.84; gap to oracle is < 0.1 points.
  • Human evaluation: PaperCoder scores 4.60/5 vs. Paper 2.76 and Abstract 2.68; ranked 1st by human judges in 88% of cases; 92% of judges rate generated repos as helpful.
  • Repos require on average only 0.81% of code lines modified to execute when errors occur.
  • Ref-based vs. ref-free evaluation correlation: \(r = 0.79\).
  • PaperBench Code-Dev (20 ICML 2024 papers, human-annotated rubrics): PaperCoder surpasses Basic Agent and Iterative Agent by substantial margins.
  • Generated repos average 14,343 tokens, 6.97 files, 35.22 functions — substantially richer than ChatDev/MetaGPT outputs.

Limitations

  • Token budget: Papers filtered to <70k tokens; very long papers or large repositories are excluded.
  • ML-domain scope: Benchmark and framework are tuned to ML papers; generalization to other scientific domains is untested.
  • LLM-as-judge bias: Reference-based and reference-free evaluations rely on o3-mini-high, which may favor certain code styles or share training signal with the generation model.
  • No direct baselines: No prior system specifically targets paper-to-code; comparisons are to systems designed for natural-language software requirements, not academic papers.
  • Sequential generation bottleneck: Coding stage generates files one-by-one in order; errors in early files propagate to all dependents with no backtracking mechanism described.

Relevance to Agentic AI / LLM Agents

PaperCoder is a concrete instantiation of multi-agent pipeline decomposition — a pattern central to agentic AI — applied to a high-value scientific task. The three-agent orchestration (planner → analyzer → coder), where each agent's output is a structured artifact consumed by the next, exemplifies how role specialization and explicit inter-agent handoffs can overcome the context-length and consistency failures of single-LLM approaches. The work also contributes to the growing literature on LLM agents as autonomous scientific workers, directly complementing benchmarks like PaperBench that assess agentic reproducibility. For researchers tracking agentic AI, the staged-decomposition architecture and the reference-free LLM-judge evaluation setup are broadly reusable patterns for tasks where ground truth is scarce.