Skip to content

Enhancing Automated Paper Reproduction via Prompt-Free Collaborative Agents

🕒 Published (v1): 2025-12-02 14:24 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes a prompt-free collaborative agent framework that wraps any multi-step paper-to-code workflow with automatic verification and refinement — no hand-crafted evaluation prompts required. A verification agent checks each workflow output against the step's original system prompt, and a refinement agent revises the output using the same system prompt as guidance. Integrated into Paper2Code, the framework achieves ~15% and ~13% improvement on PaperBench Code-Dev and Paper2CodeBench respectively.

Problem

Existing paper-to-code frameworks (e.g., Paper2Code) decompose reproduction into sequential stages (planning, coding) but lack systematic output validation between stages, allowing early errors to propagate. Self-Refine-style approaches address this but require manually designed evaluation and refinement prompts per step — 8 hand-crafted prompts for 4 planning artifacts in Paper2Code alone — which overfits to specific datasets and limits scalability.

Method

The framework introduces two agents that operate at each workflow step \(i\):

Verification agent \(V\): Given paper content \(P\), the step's system prompt \(S_i\), and current output \(O_i\), it produces a structured JSON review report: $\(R_i = V(O_i, S_i, P)\)$ where \(R_i = \{\text{completeness summary},\ \text{missing information}\ [m_1,\ldots,m_k],\ \text{action items}\ [a_1,\ldots,a_k]\}\). The key insight: \(S_i\) already encodes the full generation specification, so it doubles as the verification rubric — no separate rubric design needed.

Refinement agent \(F\): Takes \((O_i, R_i, S_i, P)\) plus all previously refined outputs \(\{O_j^*\}_{j<i}\) to produce an improved output: $\(O_i^* = F(O_i, R_i, S_i, P)\)$ The agent preserves correct parts of \(O_i\), addresses each item in \(R_i.\text{missing information}\), and maintains consistency with prior refined artifacts.

Applied to Paper2Code's planning stage (Overall Plan → Architecture Design → Logic Design → Configuration) in dependency order, and to the coding stage following the Logic Design's dependency graph. Both stages reuse the same two-agent pattern with zero stage-specific prompt engineering.

Key Contributions

  • Identification that original system prompts already contain sufficient specification to serve as both verification criteria and refinement guidance, enabling truly prompt-free self-refinement.
  • A two-agent (verify + refine) wrapper that plugs into existing multi-step workflows without modifying them.
  • First work applying prompt-free collaborative agents to multi-agent output refinement in automated paper reproduction.
  • Demonstration that manual prompt engineering in Self-Refine causes dataset-specific overfitting; prompt-free approach generalizes across distributions.

Results

PaperBench Code-Dev (20 ICML 2024 papers, GPT-4.1 vs. Paper2Code baseline): - Auto-plan optimized: +6.01% avg improvement, 80% win rate - Auto-code optimized: +9.53% avg improvement, 80% win rate - Auto-plan & code combined: +15.25% avg improvement, 85% win rate, final score 0.786 - Self-Refine (baseline): −3.96% (degraded performance, 50% win rate) - RePro (o3-mini-high, 5 iterations): +16.29% but final score only 0.614 — the proposed method matches gain in 1 iteration (5× compute reduction)

Paper2CodeBench (90 papers from ICLR/ICML/NeurIPS 2024, LLM-judge 1–5 scale): - Baseline Paper2Code: 3.84 avg - Auto-plan & code combined: 4.34 avg (+0.50) - Self-Refine: 4.05 avg (+0.21, but overfitted to this dataset)

Per-paper analysis: planning optimization outperforms Self-Refine on 16/20 PaperBench tasks; Self-Refine shows negative improvements on 10 tasks (worst: −39.6%).

Limitations

  • Verification priorities high-level completeness over fine-grained implementation details (hyperparameters, model loading), causing drops on 4/20 PaperBench tasks where such details dominate scoring (e.g., mechanistic-understanding: −26.6%).
  • Single-iteration refinement only; RePro's 5-iteration loop still achieves marginally higher relative improvement (+16.29% vs. +15.25%), though at 5× cost.
  • All experiments use GPT-4.1; cross-model generalization is not evaluated.
  • RePro comparison is indirect (different base model: o3-mini-high vs. GPT-4.1), limiting strict apples-to-apples benchmarking.

Relevance to Harnesses / Meta-Harnesses

This work is a direct instantiation of the meta-harness pattern: a wrapper layer that instruments an existing multi-step workflow (Paper2Code) with automatic verification-refinement loops at each stage boundary, without modifying the inner workflow's prompts or logic. The "prompt-free" design principle — reusing each step's own system prompt as the quality oracle — is a generalizable harness technique applicable to any workflow where steps are prompt-driven. It demonstrates that a thin meta-layer (two agents, zero additional prompt engineering) can substantially improve inner-workflow quality and generalize across task distributions where hand-tuned harness configurations fail, directly informing how meta-harnesses should be designed to avoid overfitting to specific task distributions.