Skip to content

AutoReproduce: Automatic AI Experiment Reproduction with Paper Lineage

🕒 Published (v1): 2025-05-27 03:15 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AutoReproduce is a multi-agent framework that autonomously reproduces AI experiment code end-to-end by mining implicit domain knowledge from a paper's cited literature via a "Paper Lineage" algorithm. It introduces ReproduceBench, a 13-paper benchmark with manually verified implementations, and outperforms baselines on both reproduction fidelity and execution accuracy metrics.

Problem

Reproducing AI experiments is labor-intensive because papers omit tacit knowledge—standard module architectures, data pipelines, training conventions—that practitioners treat as implicit. Existing LLM-based tools address only discrete subtasks (environment setup, repo refactoring) or generate code in a single pass without verifying executability, leaving end-to-end automatic reproduction unaddressed.

Method

AutoReproduce structures reproduction as a three-phase pipeline executed by two specialized agents (a Research Agent for text tasks and a Code Agent for implementation):

  1. Literature Review: The Research Agent applies hierarchical three-stage summarization (paper-level → method details → experimental settings) using MinerU for PDF→Markdown conversion. Optionally integrates visual diagram understanding.

  2. Paper Lineage: The Research Agent selects the top-\(k\) (default \(k=3\)) most relevant cited papers by analyzing citation relationships in full context, prioritizing comparison baselines. It fetches manuscripts via the ArXiv API, identifies linked code repositories, clones them via GitHub API, and has the Code Agent extract essential source files into <summary, code> tuples that serve as domain-aligned reference exemplars.

  3. Code Development: Three sequential stages—(i) Data Acquisition: generates loading/preprocessing code, infers tensor shapes and data types via mini-batch sampling to prevent runtime mismatches; (ii) Method Replication: dual-agent loop where the Code Agent generates implementation while the Research Agent validates against the method summary, providing corrective feedback; (iii) Experiment Execution: implements the full training pipeline, using early-exit mechanisms (e.g., break) for rapid pipeline dry-runs before full training. Code edits use a line-targeted EDIT N M command to minimize token overhead.

Execution occurs inside a Docker container. A sampling-based unit testing strategy validates executability throughout.

Key Contributions

  • Paper Lineage algorithm: systematically retrieves and summarizes cited literature and associated repositories to surface implicit implementation conventions.
  • AutoReproduce framework: end-to-end multi-agent pipeline producing executable reproduction code, not just syntactically valid code.
  • ReproduceBench: 13-paper benchmark spanning diverse AI subdomains (CV, NLP, time-series, medical imaging, graph learning, etc.) with manually curated, re-executed reference implementations and five evaluation metrics covering structural alignment (Paper-Level, Code-Level, Mixed-Level Align-Score) and final execution accuracy.

Results

  • AutoReproduce surpasses all baselines on PaperBench and ReproduceBench across all five metrics.
  • Achieves "substantial improvements" in both reproduction fidelity (Align-Score) and final execution performance relative to baselines including Paper2Code (Seo et al., 2025).
  • (Note: the provided text is truncated before specific numerical results tables are presented; no precise metric values are available in the supplied excerpt.)

Limitations

  • Paper Lineage defaults to \(k=3\) related papers; coverage of implicit knowledge depends on how well citations capture domain conventions.
  • Papers without public code repositories fall back to text summaries only, losing concrete implementation exemplars.
  • Benchmark covers 13 papers; generalization to highly specialized or non-standard domains is unvalidated.
  • Execution environment (Docker + common libraries) may not capture all dependency configurations required by complex papers.
  • Evaluation of alignment relies on an LLM judge (default o1), introducing potential scoring variance.

Relevance to Harnesses / Meta-Harnesses

AutoReproduce is a concrete instantiation of a multi-phase, multi-agent meta-harness: it orchestrates a Research Agent and Code Agent through sequenced phases with well-defined handoffs, validation loops, and tool integrations (ArXiv API, GitHub API, Docker, MinerU). The Paper Lineage algorithm is a knowledge-retrieval sub-harness that augments downstream agents with domain context—analogous to RAG-augmented harness designs. The sampling-based unit testing embedded within Code Development represents an intra-harness validation loop, a pattern relevant to anyone building harnesses that must guarantee output executability rather than just syntactic plausibility. ReproduceBench provides a rare evaluation substrate for measuring end-to-end harness performance on complex, multi-step code generation tasks.