Skip to content

From paper to benchmark: agentic, framework-based reproduction of under-specified methods in machine health intelligence

🕒 Published (v1): 2026-05-27 12:11 UTC · Source: Arxiv · link

Why this paper was selected

Agentic framework reproduces under-specified ML papers into benchmark-ready implementations

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces a framework-coupled agentic workflow that translates under-specified machine health / PHM papers into executable, benchmark-comparable implementations by binding paper concepts into typed slots of a shared PHM framework. Rather than generating isolated repositories, the agent produces structured artifacts with explicit assumption records, verification gates, and auditable failure reports. The approach is evaluated on 16 PHM papers and shown to substantially outperform prompt-only and framework-free baselines on execution rate and benchmark comparability.

Problem

Fewer than 1% of PHM papers release both code and data. Even when researchers attempt reproduction, implicit protocol decisions—windowing strategy, target-label construction, normalization scope, data splits—critically affect reported metrics yet are routinely omitted from papers. Existing paper-to-code systems (Paper2Code, DeepCode) emit isolated repositories with incompatible evaluation logic, making cross-paper comparison under shared conditions impossible. The bottleneck is not code generation per se but unresolved underspecification and the absence of a shared execution contract.

Method

The core idea is framework-grounded agentic reproduction: rather than generating a free-form repository, an agent maps a paper \(P\) into a shared PHM framework \(F\) via a slot-binding interface comprising six typed slot families—task, datasource, transform, sequencer, model, evaluator—yielding an artifact:

\[A = \langle c,\, C,\, \mathcal{A},\, r \rangle\]

where \(c\) is the resolved configuration, \(C\) is any minimal contract-preserving extension required for novel components, \(\mathcal{A}\) is the set of assumption records for every under-specified decision, and \(r\) is the verification/evaluation report.

The Framework-Coupled Agent (FCA) executes six ordered stages: INGEST (indexed paper representation with chunk-level spans), ANALYZE (extract task \(T\), model \(M\), config \(\kappa\); emit assumption records), MAP (resolve extracted spec into slot-binding plan), IMPLEMENT (materialize \(c\) and extensions \(C\)), VERIFY (static typechecking, dry-runs, leakage checks, sanity checks following Karpathy diagnostics: initial-loss, gradient-flow, micro-batch memorization), and REPORT (bundle artifact or auditable failure).

Each assumption record is the tuple \(a_i = \langle s_i, e_i, v_i, j_i \rangle\) (implicated slot family, paper-span evidence or absence-of-evidence, chosen value, justification). On verification failure with cause \(r.\text{cause}\), record \(a_i\) is attributable iff \(s_i \in \text{SLOTS}(r.\text{cause})\), enabling targeted repair without reconstructing the pipeline.

Key Contributions

  • Slot-binding interface that maps PHM paper concepts to six typed framework families under a fixed benchmark contract (splits, windowing, leakage policy, metrics held constant across papers).
  • Explicit assumption tracking via structured records \((s_i, e_i, v_i, j_i)\), making every inferred protocol decision auditable rather than buried in generated code.
  • Multi-layer verification pipeline (static contract checks → sanity/gradient checks → result matching) with attribution of failures back to implicated assumption records.
  • Multi-axis evaluation protocol covering: framework integration (slot completion rate), code development (LLM-as-judge on adapted PaperBench rubric), execution rate, result matching, and failure diagnosability, plus operational metrics Time2Build, Easiness2Build, Cost2Build.
  • Open-source release of skills, instructions, and workflow code for the agentic pipeline.

Results

  • Execution rate: FCA 14/16 (87.5%) vs. AiF 3/16 (18.75%), AiF-D 8/16 (50%), SA 14/16 (87.5%), DeepCode 0/16 (0%).
  • Slot-binding completion rate (all 6 families bound): FCA 9/16 (56.25%), AiF-D 9/16 (56.25%), AiF 7/16 (43.75%); SA achieves ≥50% missing bindings across all slot families.
  • FCA achieves on average +13.5% newly created binding states and −11% missing binding states vs. AiF across slot families.
  • LLM-as-judge code ratings: SA scores highest mean rating; FCA and DC score lowest (judges ChatGPT 5.4 and Kimi 2.6). This contradicts execution results (DC: 0% execution, high rating), raising doubts about LLM judge reliability for this task.
  • Explicit framework documentation (AiF → AiF-D) alone improves completion rate by 12.5 pp; staged workflow provides further gains in binding quality.

Limitations

  • Corpus is small (16 papers) spanning only two datasets (N-CMAPSS, NB14) and two task families (diagnostics, prognostics); generalizability to other PHM or ML domains is unverified.
  • Slot-binding completion rate (56.25% for FCA) remains below full coverage; two papers failed entirely due to insufficient specification or proprietary data.
  • LLM-as-judge code evaluation is shown to be unreliable here (high ratings for non-executing systems); human adjudication was limited to a five-paper subset.
  • The shared PHM framework \(F\) itself is under anonymous review, limiting independent validation of the framework-side contracts.
  • Result matching (reproduced metrics vs. paper-reported numbers) is not fully reported in the truncated text, leaving quantitative reproduction fidelity partially uncharacterized.
  • Operational costs (Time2Build, Cost2Build) are mentioned but not summarized quantitatively in the provided text.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct instance of a meta-harness pattern: an orchestrating agent that does not simply run a single pipeline but instead instantiates a domain-specific framework by reading external specifications (papers), resolving ambiguities through structured assumption records, and emitting verified, reusable artifacts. The slot-binding interface functions as a typed integration contract—analogous to the hook/adapter layer in software meta-harnesses—that enforces shared evaluation semantics across independently reproduced methods. The staged workflow (INGEST→ANALYZE→MAP→IMPLEMENT→VERIFY→REPORT) with verification gates and attributable failure reports is a production-grade harness design that researchers building automated benchmarking or evaluation pipelines can directly adapt. The explicit treatment of underspecification as a first-class harness concern (assumption records, attribution predicates) generalizes beyond PHM to any domain where input specifications are incomplete.