Skip to content

CodeDistiller: Automatically Generating Code Libraries for Scientific Coding Agents

🕒 Published (v1): 2025-11-30 21:19 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CodeDistiller is a system that automatically converts large collections of scientific GitHub repositories into vetted, executable code examples for use by Automated Scientific Discovery (ASD) agents via Code-RAG retrieval. It eliminates the need for manual code library construction by running an LLM-driven pipeline that classifies repository files, generates Python examples, and iteratively debugs them in a cloud container. Evaluated on 250 materials science repositories, the best model (Claude Sonnet 4.5) achieves 74% successful distillation, and downstream ASD agents augmented with the generated library outperform baselines across accuracy, completeness, and scientific soundness.

Problem

ASD agents that run computational experiments are bottlenecked by their parametric knowledge: they either mutate a small set of manually crafted examples (AI Scientist) or rely on expert-curated code libraries (CodeScientist). Scaling to new domains or new tools requires substantial human effort to build these libraries, capping the breadth and autonomy of automated discovery.

Method

CodeDistiller operates as a two-phase agentic pipeline:

  1. Relevant File Classification: Every file in a repository is individually classified by a cheap LLM (e.g., Haiku 4.5, GPT-5-mini) into coarse types (code, documentation, scripts, data, other) with fine-grained subcategories and a relevance score (1–5), plus metadata about GPU requirements and configuration needs.

  2. Code Generation & Iterative Debugging: The highest-ranked files and a repository purpose summary are fed to a more capable LLM (Claude Sonnet 4.5, GPT-5, or GPT-OSS-120B), which generates (a) executable Python code, (b) pip requirements, (c) a Conda bash runscript, and (d) structured metadata (description, inclusion/exclusion criteria, compute requirements). The generated code is executed in an Ubuntu cloud container; an LLM-as-a-judge determines success or identifies errors. On failure, the code is reflected and re-executed up to 8 iterations. Successful examples are stored in a Code-RAG library for downstream ASD agents.

Key Contributions

  • CodeDistiller system: End-to-end pipeline for automatically distilling GitHub repositories into debugged, runnable domain-specific code examples at scale.
  • Materials science evaluation: Benchmarked on 250 repositories, with best model (Claude Sonnet 4.5) achieving 74% automatic success and 74% manual expert-validated correct functionality on a 50-repo subsample.
  • Downstream ASD improvement: CodeScientist augmented with a CodeDistiller library is preferred by domain experts over the baseline (generic examples only) in >50% of 50 experimental tasks across accuracy, completeness, and soundness dimensions.
  • LLM-as-a-judge characterization: Moderate agreement (Cohen's κ = 0.62–0.77) between LLM-as-a-judge and domain expert ratings on A/B downstream evaluation, with accuracy approaching strong agreement (κ = 0.77) and completeness being the weakest (κ = 0.62).

Results

  • Distillation success (automatic/LLM-as-a-judge):
  • GPT-OSS-120B: 61.6%; GPT-5: 70.4%; Claude Sonnet 4.5: 75.6%
  • Distillation success (domain-expert manual, 50-repo subsample, correct functionality):
  • GPT-OSS-120B: 25.9%; GPT-5: 60.5%; Claude Sonnet 4.5: 74.1%
  • LLM-as-a-judge over-counts success: GPT-OSS-120B reports 62% success vs. only 26% by expert—a 2.4× overestimate; Claude Sonnet 4.5 shows minimal gap (76% auto vs. 74% manual).
  • Runtime: Successful examples take 13–21 minutes and ~1.9–2.4 debug iterations on average; unsuccessful runs exhaust up to 8 iterations.
  • Cost: Average cost per successful example ranges from $0.09 (GPT-OSS-120B) to $2.26 (Claude Sonnet 4.5).
  • Downstream A/B (domain expert preference, 50 tasks):
  • CodeDistiller-augmented agent preferred for accuracy in ~56% of cases (baseline: ~20%, tie: ~24%)
  • CodeDistiller preferred for soundness in ~58% of cases (baseline: ~22%, tie: ~20%)
  • CodeDistiller preferred for completeness in ~40% of cases (baseline: ~18%, tie: ~42%)

Limitations

  • Domain expert evaluation is time-bounded: Manual inspection provides a proxy assessment, not exhaustive verification against known-correct outputs; judging true correctness would require generating gold-standard tests for each repository.
  • Repository filtering noise: ~50% of automatically identified "materials science" repositories are unrelated but happen to import domain-adjacent libraries; domain expert subsample selection was used to mitigate this.
  • Single-domain evaluation: All experiments are in materials science; performance and cost characteristics may differ substantially in other domains with different software/data availability profiles.
  • Unsuccessful runs are costly: Failed attempts iterate to the 8-iteration hard limit, incurring disproportionate API cost and time relative to successful runs.
  • LLM-as-a-judge reliability is mixed: Strong agreement for accuracy/soundness but weaker for completeness; separate calibration needed per domain and per evaluation model.
  • Purpose-built vs. general-purpose tradeoff unresolved: The paper does not compare CodeDistiller against general-purpose agents like Claude Code or Codex under controlled conditions.

Relevance to Agentic AI / LLM Agents

CodeDistiller directly addresses a core bottleneck in tool-augmented agentic systems: the cold-start problem of building domain-specific tool/code libraries without human curation. By automating Code-RAG library construction, it enables ASD agents to generalize to new scientific domains without manual effort, which is a critical scalability requirement for long-horizon autonomous agents. The iterative generate-execute-debug loop instantiated here is a reusable design pattern for any agentic system that must produce verified, runnable artifacts from under-specified inputs. The LLM-as-a-judge characterization (κ ≈ 0.62–0.77 vs. domain experts) is directly applicable to evaluating agent output quality at scale in specialized domains where ground truth is expensive to obtain.