AI scientists produce results without reasoning scientifically¶

🕒 Published (v1): 2026-04-20 20:23 UTC · Source: Arxiv · link

Why this paper was selected

LLM-based scientific agents achieve results but violate epistemic reasoning norms

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Across 25,000+ runs of LLM-based scientific agents spanning 8 domains, the base model—not the scaffold—determines both performance and epistemic behavior, accounting for 41.4% of explained variance versus 1.5% for the scaffold. Agents routinely ignore evidence (68% of traces), rarely revise beliefs via refutation (26%), and never update beliefs in 71% of traces—patterns that persist even under extreme scaffold interventions injecting near-complete successful trajectories. The authors conclude that scaffold engineering cannot repair reasoning failures that are properties of the base model.

Problem¶

Existing benchmarks for scientific agents measure task completion (did the agent get the right answer?) but not the epistemic process (did the agent reason correctly to get there?). This gap means outcome-based evaluation cannot detect failures in scientific reasoning—evidence non-uptake, untested hypotheses, absent belief revision—that undermine the justifiability and reliability of AI-produced scientific knowledge, especially in hypothesis-driven inquiry where disciplined evidence use is essential.

Method¶

Agents are modeled as language decision processes \(\pi_\text{overall} = f(\pi_\text{llm}, \pi_\text{scaffold})\), factoring a base-model policy from a scaffold policy. Three frontier models (GPT-4o gpt-4o-2024-08-06, Claude Sonnet 4.5 claude-sonnet-4-5-20250929, GPT-OSS gpt-oss-120b) are paired with two scaffold architectures (ReAct and structured tool-calling) across 8 scientific domains ordered by epistemic demand: workflow-construction (molecular simulation, surface construction, ML property prediction, AFM execution), strategic reasoning (retrosynthetic planning), and hypothesis-driven inquiry (spectroscopic structure elucidation, inorganic qualitative analysis, circuit inference).

Performance analysis uses Item Response Theory (IRT) to estimate latent knowledge \(\tilde\theta_K\) and reasoning \(\tilde\theta_R\) abilities from domain-specific diagnostic QA batteries, then combines these with environment/scope/scaffold parameters in a Bayesian latent-factor model (selected via PSIS-LOO cross-validation across M1–M8 specifications) to decompose variance.

Epistemological analysis annotates each step in agent traces with epistemic operations—hypothesis (H), evidence (E), test (T), judgment (J), update (U), commitment (C)—and represents their dependencies as directed graphs. Productive patterns (refutation-driven revision, convergent multi-test evidence) and anti-patterns (untested claims, evidence non-uptake, premature commitment) are then measured across 25,000+ runs.

A trace intervention experiment injects partial trajectories from prior successful or failed runs at varying extents (early steps through \(n-1\)) to probe whether scaffold-level context can overcome reasoning failures.

Key Contributions¶

Decomposition showing base model accounts for 41.4% of performance variance; scaffold contributes only 1.5%
Epistemological annotation taxonomy and graph-based framework for classifying agent reasoning traces (H/E/T/J/U/C nodes, directed dependency edges)
Catalog of productive reasoning motifs and anti-patterns with prevalence measurements across 25,000+ runs across 8 domains
Trace intervention methodology revealing that near-complete trajectory injection helps only workflow tasks (mean log-prob \(-0.20\)), not hypothesis-driven tasks (mean log-prob \(-0.27\))
Demonstration that \(\text{Pass}^\wedge k\) (all \(k\) trials succeed) falls below 0.05 by \(k = 4\)–6 in hypothesis-driven domains, even under success-trace intervention
Open interactive benchmark (CORRAL) with standardized tool interfaces and 90+ tools

Results¶

Variance decomposition: reasoning ability 41.4%, environment–scope interaction 30.1%, scaffold 1.5%, verbosity 0.1%
Evidence non-uptake: 68% of all traces; 82% in workflow domains, 66% in strategic, 60% in hypothesis-driven
Beliefs never updated: 71% of traces
Refutation-driven belief revision: only 26% of traces
Untested claims: 53% overall, 63% in hypothesis-driven domains
Convergent multi-test evidence: 7% of traces; ≤10% in every domain group
Top configurations in workflow-construction domains approach performance ceilings; best configuration falls below 60% in Inorganic Qualitative Analysis and Spectroscopic Structure Elucidation at broadest scope
Reasoning gap across models: largest separation in Inorganic Qualitative Analysis (\(>5.3\) standard units in IRT reasoning ability between strongest and weakest model)
Trace intervention: in workflow domains, 1–2 injected successful steps exceed baseline; in hypothesis-driven domains, gains appear only at steps \(n-2\) or \(n-1\)
\(\text{Pass}^\wedge k\) decay: drops below 0.05 by \(k = 4\)–6 for Spectroscopic Structure Elucidation and Inorganic Qualitative Analysis under baseline; near-complete success-trace injection does not prevent this decay

Limitations¶

Only three frontier models tested; open-source model diversity is limited (one open-source: GPT-OSS 120B)
Epistemological annotation itself uses Claude Sonnet 4.5 as annotator, introducing potential circularity and model-specific annotation biases
All eight domains are chemistry/materials-science-adjacent; generalization to other scientific fields (biology, physics, social sciences) is not demonstrated
Trace interventions inject only single-source trajectories; more complex multi-source or hierarchical interventions are unexplored
The latent-factor model predicts task success probability but does not directly optimize or prescribe training interventions
Paper text is truncated; full discussion of limitations and confounds from Section A.6 onward is not available

Relevance to Harnesses / Meta-Harnesses¶

This paper is directly relevant as a rigorous scaffold evaluation meta-study: it systematically disentangles what scaffold architecture (ReAct vs. structured tool-calling) contributes to agent behavior versus the base model, finding scaffold contribution nearly negligible (1.5% variance). For harness designers, the key finding is that orchestration logic, prompt formatting, tool routing, and observation injection—the typical levers of a harness—cannot compensate for base-model reasoning deficits; the paper provides a quantitative upper bound on what scaffold engineering can achieve. The CORRAL framework functions as a meta-harness in its own right: a common implementation layer with standardized tool interfaces, 90+ tools, and graded task scopes that can accommodate new models, scaffolds, and metrics, making it a reusable evaluation substrate for future harness research. The trace intervention methodology also models a form of in-context scaffolding (injecting prior successful runs), directly probing the mechanism by which harness context injection helps or fails.