Skip to content

Read the Paper, Write the Code: Agentic Reproduction of Social-Science Results

🕒 Published (v1): 2026-04-23 17:59 UTC · Source: Arxiv · link

Why this paper was selected

LLM agents reproduce social science results from methods text alone; generalization test

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

The paper builds a multi-step agentic pipeline that reproduces empirical social-science results from a paper's methods description and raw data alone—without any author-provided code. Evaluated on 48 human-verified-reproducible papers, frontier LLM agents recover the correct sign for 78–91% of regression coefficients. Failure attribution shows underspecification in papers is a larger source of error than agent mistakes.

Problem

Prior agentic reproducibility work gives agents the original analysis code, making the task closer to re-execution than re-implementation. The open question is whether a paper's methods section is sufficiently detailed for an agent to reimplement the analysis from scratch—the standard expected of a human reader. No prior system evaluated this under strict information isolation with deterministic (non-LLM-judge) evaluation for social science.

Method

A four-step pipeline enforces end-to-end information isolation:

  1. Extraction: GPT-5-mini parses the paper PDF (rendered as images) to produce (a) a structured methods description with numerical results stripped and verified absent, (b) a blinded results-table template with all cell values set to null, and (c) a minimal data manifest pointing to least-preprocessed files in the reproduction package.

  2. Reimplementation: An autonomous LLM agent—operating in a sandboxed workspace with no access to the original code, PDF, or numerical results—reads only the extracted methods and blank table templates, explores the provided data, and writes one Python script per table that fills in the template.

  3. Evaluation: Reproduced cell values are compared deterministically to originals. Regression coefficients receive letter grades (A–F) based on percent absolute deviation; sign mismatch maps to E; missing output maps to F. Grades aggregate at the table then paper level. A normalized metric \(|\hat{\beta} - \beta| / \text{SE}\) measures statistical agreement.

  4. Explanation: A separate LLM agent inspects mismatched cells by reading both the original code and the agent's generated script, attributing errors to one of four buckets: Missing Data, Paper vs. Code, Paper vs. Methods Extraction, or Methods Extraction vs. Agent.

A two-stage compliance audit (deterministic regex path scan + GPT-5.4-mini severity rating) and a hardcoding audit verify that agents genuinely compute results rather than retrieve or memorize them.

Key Contributions

  • First agentic reproduction benchmark for social science that withholds all author code, providing only paper text and raw data.
  • Strict information-isolation architecture with automated guardrail and hardcoding audits.
  • Deterministic cell-level grading scheme with a normalized \(|\Delta\beta|/\text{SE}\) metric, avoiding LLM-as-judge ambiguity.
  • Four-category error attribution step distinguishing human (underspecification, missing data) from agent (extraction loss, noncompliance) failure modes.
  • Comparative evaluation of 4 LLMs Ă— 4 agent scaffolds (Claude Code Opus 4.6, Codex CLI GPT-5.3/5.4, SWE-Agent GPT-5.4/GLM-5, OpenCode GPT-5.4/GLM-5) on 48 papers from top economics and political science journals.

Results

  • Sign agreement on reproduced regression coefficients: 78% (SWE-Agent GPT-5.4) to 91% (OpenCode GPT-5.4); naive "guess positive" baseline is 68%.
  • Best agent (OpenCode GPT-5.4): reproduced coefficients fall within the 95% CI (\(|\Delta\beta|/\text{SE} \leq 1.96\)) >80% of the time.
  • For the three best agents: sign agreement >85% and within-95%-CI rate >70%.
  • Paper-level completion: 92–100% of papers yield usable results; table completion 82–97%; cell completion 52–72%.
  • Best overall cell completion: Claude Code Opus 4.6 (72%); worst: SWE-Agent GLM-5 (52%).
  • Best reproduction accuracy: OpenCode GPT-5.4, attributable to substantially higher token usage and cost.
  • Scaffold matters more than model tier: GPT-5.4 on OpenCode outperforms GPT-5.4 on Codex CLI or mini-SWE-Agent.
  • Failure attribution: the majority of errors trace to paper underspecification, not agent errors.
  • Leakage probe (papers published after August 2025 knowledge cutoff) shows no significant performance drop, suggesting memorization is not the primary driver.

Limitations

  • Restricted to tables; figure-level comparison requires VLMs whose reliability for this task is uncertain.
  • Dataset of 48 papers is small and biased toward fully-reproducible papers (I4Replication-verified), excluding papers that fail even with code access.
  • Original code is predominantly Stata/R; agents reimplement in Python, introducing potential numerical divergence unrelated to methods understanding.
  • Methods extraction quality is a bottleneck: incomplete extraction propagates errors to reimplementation, and the extraction LLM is not the agent under evaluation.
  • ~8% of tables excluded due to extraction issues; landscape/multi-page tables show occasional decimal-place extraction errors.
  • Data leakage probe uses only 10 papers from a single journal and cannot rule out partial memorization.
  • Token budget and API cost scale with quality (OpenCode GPT-5.4's superiority is tied to much greater token usage), making the approach expensive.

Relevance to Harnesses / Meta-Harnesses

This paper is a concrete instantiation of a multi-step meta-harness: it composes extraction, reimplementation, evaluation, and explanation sub-agents into a deterministic outer pipeline with explicit information-flow controls between stages. The compliance audit subsystem (regex path scan + LLM severity rating + hardcoding detector) is itself a meta-harness component that monitors and validates inner-agent behavior—directly relevant to harness designers who need to enforce isolation invariants across agent hand-offs. The four-stage error attribution step models how a meta-harness can attribute failures to specific pipeline nodes rather than treating the system as a black box. The comparative evaluation of four agent scaffolds as interchangeable "execution backends" within a fixed outer harness provides empirical data on how scaffold choice affects end-to-end pipeline performance.