NewtonBench: Benchmarking Generalizable Scientific Law Discovery in LLM Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Rigorous benchmark addressing trilemma in scientific law discovery for LLM agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

NewtonBench is a benchmark of 324 scientific law discovery tasks across 12 physics domains that evaluates LLM agents on interactive, agentic discovery rather than static curve-fitting. It resolves the scalability/relevance/memorization trilemma via counterfactual law shifts on canonical physical laws, and requires agents to actively probe simulated systems to uncover hidden equations. Evaluation of 11 frontier LLMs reveals strong but fragile capability: performance collapses with system complexity and noise, and code assistance paradoxically hurts the most capable models.

Problem

Existing scientific law discovery benchmarks (e.g., AI Feynman, LLM-SR, LSR-Transform) share a fundamental methodological trilemma: they cannot simultaneously achieve scientific relevance, scalability, and resistance to memorization. Worse, all prior benchmarks frame discovery as passive, static function fitting over a pre-given dataset—an impoverished proxy for the real scientific process of actively experimenting on complex systems with confounding variables.

Method

Counterfactual law shifts mutate canonical physical laws (e.g., Newton's gravitation, Coulomb's law) by applying operator or exponent mutations on their expression trees \(T\), generating physically novel but conceptually grounded variants. Dimensional consistency is preserved by adjusting the embedded physical constant. Three difficulty tiers (Easy/Medium/Hard) are defined by the cumulative number of mutations (1–2, 3–4, 5–6).

Each shifted law is embedded inside one of three model systems of increasing extrinsic complexity: - Vanilla Equation: \(M = (f_\text{target})\) only. - Simple System: \(f_\text{target}\) embedded in a small multi-equation model. - Complex System: \(f_\text{target}\) embedded in a larger model with more confounding equations.

Agents interact with the virtual environment via a <run_experiment> tool that acts as an interactive black-box: the agent proposes input assignments \(V_M\) and receives outputs \(Y_M\), iteratively building evidence to deduce \(f_\text{target}\). An optional code assistance setting provides a Python interpreter (<python> tags) for regression and hypothesis testing. Evaluation uses two metrics: Symbolic Accuracy (binary structural equivalence checked by an LLM-as-a-judge achieving 98.3% agreement with humans) and RMSLE: $\(\text{RMSLE} = \sqrt{\frac{1}{N}\sum_{i=1}^{N}\left(\log(\hat{y}_i+1)-\log(y_i+1)\right)^2}\)$

Key Contributions

• NewtonBench dataset: 324 tasks = 108 counterfactually shifted laws \(\times\) 3 system complexity levels, spanning 12 physics domains (gravitation, Coulomb, Snell, Ampère, Fourier, Hooke, radioactive decay, harmonic motion, Malus, Bose-Einstein, acoustic velocity, heat transfer).
• Counterfactual law shift methodology that simultaneously achieves scalability, scientific relevance, and memorization resistance.
• Agentic interactive evaluation paradigm: shifts benchmark from passive function fitting to active multi-turn model discovery with a black-box experiment tool.
• Two orthogonal difficulty axes: intrinsic (equation mutation depth) and extrinsic (model system complexity), enabling fine-grained capability profiling.
• Empirical finding of the code-assistance paradox: tool use degrades capable models by inducing premature exploitation over exploration.
• LLM-as-a-judge symbolic equivalence checker (98.3% human agreement).

Results

• Non-thinking models fail: GPT-4.1-mini (5.1% SA), GPT-4.1 (5.8% SA), DeepSeek-V3 (8.9% SA) all score below 10% average SA; performance collapses outside the simplest Vanilla/Easy cell.
• Top reasoning models (vanilla agent): GPT-5 leads at 75.9% SA (RMSLE 0.249); Gemini-2.5-pro second at 65.4% SA (RMSLE 0.354).
• Complexity collapse: In the hardest cell (Complex System, Hard law), GPT-5 falls to 40.3% SA and Gemini-2.5-pro to 16.7%; all other reasoning models fall below 6%.
• Code assistance paradox: For weak models (SA < 40%), code tool substantially helps (e.g., GPT-4.1-mini jumps from 5.1% to 14.7%). For strong models (SA ≥ 40%), code slightly hurts (GPT-5: 75.9% → 73.3%; Gemini-2.5-pro: 65.4% → 63.7%), traced to premature exploitation of local numerical fitting at the expense of global symbolic search.
• Noise sensitivity: Even minimal observational noise causes sharp drops in symbolic accuracy (detailed in §4.2, figures omitted from excerpt).

Limitations

• Only 12 canonical seed laws; generalization to chemistry, biology, or social-science laws is untested.
• 108 shifted laws is a comparable but still moderate corpus; the benchmark size is bounded by expert curation requirements for scientific plausibility.
• The virtual environment simulates deterministic, noise-free systems by default; real laboratory stochasticity is handled only as an add-on noise analysis.
• LLM-as-a-judge for symbolic equivalence, while high-agreement, introduces a potential failure mode if the judge model itself is being evaluated.
• Results are sensitive to prompt scaffolding choices (tool syntax and objectives are given; strategy is not), making cross-benchmark comparison non-trivial.

Relevance to Agentic AI / LLM Agents

NewtonBench directly operationalizes a core agentic loop—hypothesis formation, active experiment design, evidence integration, law submission—making it a natural testbed for evaluating scientific reasoning agents beyond simple question-answering. The discovery of the code-assistance paradox (capable agents over-exploiting a tool and satisficing on suboptimal solutions) is a concrete, quantified instance of the exploration-exploitation tension central to agentic planning research. The two-axis difficulty design (law complexity × system complexity) provides a principled stress-test for agents that must disentangle confounders through sequential tool use, directly relevant to multi-step tool-augmented agent evaluation. The benchmark's emphasis on generalizable, noise-robust discovery—rather than memorized pattern matching—raises the bar for what counts as genuine agentic scientific capability.