Skip to content

What Makes AI Research Replicable? Executable Knowledge Graphs as Scientific Knowledge Representations

🕒 Published (v1): 2025-10-20 17:53 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces xKG (Executable Knowledge Graph), a paper-centric knowledge base that extracts hierarchical technique-code pairs from scientific literature to help LLM agents replicate AI research. xKG is a pluggable module—not a new agent—that slots into existing agent frameworks and boosts replication scores by up to +10.90% (PaperCoder + o3-mini) on PaperBench Code-Dev.

Problem

LLM agents tasked with reproducing AI research fail because (1) critical implementation details are implicit and scattered across papers, code, and configuration; (2) RAG over raw text misses latent technical specifics; and (3) prior knowledge-reuse approaches are coarse-grained and lack executable grounding, causing agents to produce scaffold-level dummy code rather than working implementations.

Method

xKG models a paper corpus as a hierarchical multi-relational graph \(\mathcal{G} = (\mathcal{N}, \mathcal{E})\) with three node types and two edge types:

  • Paper Node \(n_p = (M_p, \{n_t\}_i, \{n_c\}_j)\): metadata, technique nodes, code nodes.
  • Technique Node \(n_t = (D_t, \{n'_t\}_k)\): a self-contained academic concept with optional sub-nodes, linked by Structural Edges \(e_\text{struct}\).
  • Code Node \(n_c = (\sigma, \tau, \delta)\): executable implementation \(\sigma\), test script \(\tau\), documentation \(\delta\), linked to techniques via Implementation Edges \(e_\text{impl}\).

Construction is a three-step automated pipeline per target paper: 1. Technique Extraction: o4-mini deconstructs the paper into a technique hierarchy; RAG (text-embedding-3-small) enriches each node definition \(D_t\). 2. Code Modularization: Per-technique embedding retrieval over the associated GitHub repo; o4-mini synthesizes candidate Code Nodes; an iterative self-debugging loop enforces executability. 3. Knowledge Filtering: Technique nodes for which no code was retrieved are pruned—only techniques grounded in executable code survive.

At inference, agents query xKG in two stages: high-level planning uses Paper Nodes (technique overview); low-level implementation retrieves (Technique, Code) pairs, gated by an o4-mini Verifier for technical relevance.

Key Contributions

  • xKG formal graph model linking scientific concepts to executable code at multiple granularities.
  • Fully automated, paper-aware construction pipeline (corpus curation → KG construction → executability verification) averaging ~$0.73/paper.
  • A self-debugging loop that raises Code Node executability from 52.38% → 100%.
  • A pluggable integration protocol compatible with ReAct agents (tool calls) and fixed-workflow agents (pluggable components).
  • Demonstrated self-evolution: xKG expands continuously as new target papers are introduced, without manual intervention.

Results

  • PaperCoder + o3-mini: +10.90% average replication score (42.31% → 53.21%) on PaperBench Code-Dev lite.
  • IterativeAgent + o3-mini: +8.20% average (27.02% → 35.22%).
  • BasicAgent + o3-mini: +7.15% average (19.35% → 26.50%).
  • Ablation (PaperCoder + o3-mini, Table 2): removing Code Nodes costs −4.56%; removing Paper Nodes costs −2.13%; removing Technique Nodes costs −1.05%.
  • Code quality study (Figure 2): Raw code already helps; LLM-rewritten-but-unverified (+Rewrite) degrades performance versus verified; the Verifier step improves score by 5.75–8.20 pp.
  • Human evaluation (Table 3): 89.44% of Technique Nodes are self-contained; 100% of Code Nodes are executable; 74.51% of Tech-Code pairs are exactly matched.
  • Results are highly paper-dependent: e.g., BasicAgent + o3-mini yields +24.26% on MU-DPO but −0.15% on FRE.

Limitations

  • Evaluated only on PaperBench Code-Dev lite subset due to funding constraints; full benchmark results unavailable.
  • Requires at least some existing reference papers; fails for truly novel domains with no related literature.
  • Transfer to tasks beyond paper replication (e.g., ML engineering benchmarks) is proposed but not demonstrated.
  • High variance in PaperBench scores makes individual-paper results noisy; best@3 mitigates but does not eliminate this.

Relevance to Harnesses / Meta-Harnesses

xKG is a canonical pluggable knowledge harness: it wraps an automated construction pipeline (corpus curation → graph build → verification) that can be dropped into any agent framework without modifying the agent itself. This pattern—building a reusable, self-evolving artifact that amplifies downstream agent harnesses—is directly analogous to how meta-harnesses manage shared context, tool routing, and retrieval augmentation across multiple pipeline stages. The two-stage integration protocol (planning context vs. implementation retrieval) mirrors the plan/execute split common in meta-harness designs, and the LLM Verifier acting as a final quality gate is a textbook example of a harness-level filter interposed between retrieval and generation.