Coding-agents can replicate scientific machine learning papers¶

🕒 Published (v1): 2026-07-02 13:11 UTC · Source: Arxiv · link

Why this paper was selected

Coding agents replicating SciML papers; concrete agentic capability benchmark

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Paper-replication is a coding-agent skill that operationalizes scientific paper replication as a target-level evidence contract: each computational claim becomes a tracked target requiring a linked evidence bundle before it can be marked matched. Evaluated on 12 independent agent runs across 4 scientific ML papers, all 12 workspaces reach completion and all 158 recorded targets are matched. The key insight is that completion is a workspace state verified by external checks, not the agent's self-reported final message.

Problem¶

Prompting a coding agent to replicate a paper is unreliable: agents stop early, lose track of remaining claims, count copied paper figures as generated outputs, or use substitute methods that produce plausible-looking results without faithfully implementing the paper. No prior workflow defines paper replication as a target-level evidence contract in a persistent workspace—existing benchmarks rely on author rubrics, masked codebases, or reproduction packages rather than paper materials alone.

Method¶

Paper-replication is implemented as a coding-agent skill (reusable task instructions + workspace utilities). The workflow proceeds in five stages, each producing workspace records that are then validated by external checks:

Source inspection: The agent traverses the LaTeX source tree ($P$), records a file inventory, and hashes all paper-provided assets. Hash checks later detect whether generated outputs were copied from paper materials.
Target definition: The agent records a finite target set $T = \{t_j\}_{j=1}^J$ in a reproduction matrix, one entry per computational claim (figure, table entry, scalar, distribution, structural result). A task ledger enforces one active target at a time.
Method reconstruction: Before any target can be matched, the agent writes specification files recording its restatement of paper equations, implementation plan, and recorded hypotheses for underdetermined choices (seeds, tolerances, preprocessing).
Evidence generation: For each target $t_j$, the agent produces candidate result $\hat{y}_j = F_j(D_j;\,\theta_j,\omega_j)$ and assembles an evidence bundle $$E_j = \bigl(\hat{y}_j,\, R_j,\, P_j,\, C_j,\, G_j\bigr)$$ where $R_j$ is the run record, $P_j$ is provenance (code, config, seed, paper passages), $C_j$ is comparison under the target's acceptance rule, and $G_j$ records report coverage.
Validation checks: External checks (run outside the agent) verify paper-asset separation, ledger/matrix consistency, specification completeness, run-generated provenance, comparison evidence under the pre-recorded acceptance rule, and report coverage. The completion gate passes only when every target is matched, no target remains active, and the final replication report PDF exists.

Acceptance rules are claim-specific: numeric targets use a discrepancy metric and tolerance; distributional targets compare summary statistics, quantiles, or interval coverage; structural targets check for patterns (sparsity, ordering, attractor geometry); visual targets compare rendered figures.

Key Contributions¶

Formal definition of paper replication as a target-level evidence task: each target $t_j$ must link $\hat{y}_j$ to the paper's method, a successful run, provenance, comparison, and report coverage before being marked matched.
Implementation as a coding-agent skill with a persistent replication workspace and external validation checks, addressing prompt-only failure modes via harness engineering.
A repeated-run case-study evaluation methodology measuring paper-level completion, target coverage, paper-anchored numeric fidelity, elapsed time, correction work (intermediate runs replaced before acceptance), and acceptance-rule variation across independent runs.

Results¶

Completion: All 12 independent runs across 4 papers (PIFT, PINN-I, PINN-II, SINDy) pass the completion gate.
Coverage: All 158 recorded targets are matched with report coverage across the 12 workspaces.
Run variation (observed, not aggregated as single numbers): repeated runs differ in how targets are decomposed, in numeric fidelity to paper-reported values, in elapsed replication time, in number of intermediate executions replaced before final evidence is accepted, and in the acceptance-rule types recorded per target.
No paper-provided asset copying was observed across any of the 12 runs (hash checks did not fire).

Limitations¶

Evaluation covers only 4 papers; generalization to papers with more complex compute requirements, external experimental data, or opaque codebases is untested.
Hash checks do not prevent every adversarial or transformed reuse of paper-provided materials.
Exact numerical agreement is not required; acceptance rules are agent-assigned and claim-specific, introducing judgment variation across runs.
The workflow requires a compute environment capable of running the relevant experiments and requires datasets to be included with paper materials or generatable synthetically.
Numeric fidelity to source papers varies across runs—completion does not imply close quantitative agreement with the original results.
The study uses a no-author-code policy; behavior under permissive author-code policies is not evaluated.

Relevance to Agentic AI / LLM Agents¶

This paper is a direct case study in harness engineering—the design principle that agent reliability for long-horizon tasks comes from modifying the environment (persistent workspace, external validation checks) rather than from prompt instructions alone. The workspace-as-ground-truth pattern (the workspace, not the chat transcript, is the record of the run) directly addresses known failure modes in long-horizon agent tasks: specification gaming, early termination, and self-reported false completion. The one-active-target-at-a-time task ledger and external completion gate are concrete mechanisms for enforcing progress tracking in multi-step agentic workflows, with broad applicability beyond paper replication to any agent task requiring auditable, evidence-linked output.