Skip to content

Articulating Assumptions in AI-Generated Scientific Analyses through Task Decomposition

🕒 Published (v1): 2026-07-07 02:40 UTC · Source: Arxiv · link

Why this paper was selected

Task decomposition to surface and articulate assumptions in LLM-generated scientific analysis pipelines

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces a multi-agent framework for generating, executing, and auditing LLM-produced scientific analysis code, with an emphasis on surfacing hidden assumptions and ambiguities in natural-language specifications. A dedicated tracing agent reconstructs an "as-implemented specification" from generated code, and a critique agent compares it against the original intent. The modular design enables smaller models (14B–32B) to execute a workflow previously requiring 70B-scale models.

Problem

LLM-generated scientific code suffers from two distinct uncertainty sources: (1) ambiguous natural-language specifications that leave computational choices implicit, and (2) stochastic generation variability producing semantically incorrect but syntactically executable code. In scientific domains like high-energy physics, executability is insufficient—what was computed, on which data, under which assumptions, must all be auditable. Existing single-prompt approaches (e.g., CoLLM) provide no structured mechanism for exposing or comparing intended vs. implemented behavior.

Method

The framework decomposes code generation into six sequential LLM stages, each producing inspectable intermediate artifacts:

  1. Domain initialization: A swappable domain package supplies generation profile, helper registry, helper-selection policy, critique context, and I/O contract—decoupling domain knowledge from orchestration logic.
  2. Helper selector: An LLM agent reads the task card and helper registry metadata, recommending relevant pre-built functions; the selected helpers are merged into an enriched task card to constrain generation.
  3. Code generator: Produces a Python analysis script from the enriched task card and domain profile.
  4. Executor + fixer/regenerator: Runs code as a subprocess; on failure, a fixer agent patches using runtime diagnostics; if repair fails after \(N\) retries, a regenerator (temperature 0.3) produces a fresh implementation.
  5. Tracer: Receives the final executable and original task card; reconstructs an as-implemented specification by tracing dependency chains (e.g., "A computed as B → C → D") for each requested quantity—without domain assets, making it domain-agnostic.
  6. Critique: Compares the original task card against the tracer output and source code, producing a structured findings report with six finding types: (A) composite violation, (B) definition mismatch, (C) missing implementation, (D) specification ambiguity, (E) extra behavior, (F) generator-induced bias.

An optional Golden Axe Oracle module pre-processes the task card before code generation, identifying up to three high-risk ambiguous phrases, explaining possible interpretations, and suggesting rewrites. A templated variant (oraclet) uses fixed ambiguity categories for improved stability with small LLMs. All intermediate artifacts (helpers, generated code, execution logs, tracer output, critique report, oracle output) are persisted in a run directory for reproducibility and prompt improvement.

Key Contributions

  • Quantity-grounded semantic differencing: a formalization of comparing LLM-intended vs. implemented scientific computation via structured trace chains.
  • Six-stage multi-agent pipeline with explicit separation of helper selection, generation, execution-based repair, tracing, and critique.
  • Domain asset interface: a fixed-interface package pattern that makes the orchestration logic domain-agnostic; swapping lhco_domain for another package applies the full workflow to a new scientific field.
  • Golden Axe Oracle: pre-generation ambiguity detection module with both free-form and templated variants.
  • Small-model enablement: task decomposition allows Qwen-14B to execute main workflow components (Qwen-32B for robustness), versus the prior 70B requirement in CoLLM.
  • Provenance-oriented artifact store: complete run directory supports prompt tuning and audit of each generation stage.

Results

  • Validated on representative LHC Olympics (LHCO) collider physics analyses adapted from the CoLLM benchmark task cards.
  • Helper selection stability confirmed across repeated runs on benchmark task cards.
  • Tracer correctly reconstructs as-implemented specifications for LHCO analyses at the 14B–32B parameter scale.
  • Critique module successfully identifies the six defined finding categories in generated code.
  • Qwen-14B executes main workflow components "in a usable manner"; Qwen-32B shows "greater robustness and consistency."
  • Task decomposition demonstrates improved transparency and reliability relative to the prior single-prompt CoLLM approach (no aggregate numeric scores reported in the provided text).

Limitations

  • Quantitative evaluation metrics (e.g., pass@k, F1 for critique findings) are not reported in the provided text; validation is described qualitatively.
  • Tracer system prompt is "rather long" and extensively validated; users are cautioned not to modify it without re-validation—fragility at the most domain-agnostic stage.
  • Only two domain packages implemented (LHCO physics and lightweight testing); generalization to other scientific domains is architectural but not empirically demonstrated here.
  • Oracle detects at most three high-risk phrases per task card; systemic ambiguities in complex specifications may go undetected.
  • Regeneration agent uses temperature 0.3 while all others use 0; this asymmetry is not ablated.
  • Framework requires pre-curated helper registries and domain expert-authored oracle/critique contexts, creating upfront knowledge-engineering cost.

Relevance to Harnesses / Meta-Harnesses

This paper is a direct instance of a scientific meta-harness: it orchestrates multiple specialized LLM agents—helper selector, generator, fixer, regenerator, tracer, critique, oracle—under a single domain-agnostic orchestration layer, precisely the pattern that harness research examines. The domain asset interface (generation profile, helper registry, selection policy, critique context, I/O contract) formalizes how harnesses should parameterize domain knowledge without coupling it to control flow. The provenance-oriented artifact store and modular retry logic (fixer vs. regenerator escalation) are concrete harness engineering decisions with direct analogs in general-purpose agentic frameworks. For researchers tracking this topic, the paper provides a worked example of decomposing a complex analytical task into auditable, inspectable sub-tasks where the harness structure itself improves both reliability and model-size requirements.