Skip to content

Can Language Models Discover Scaling Laws?

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; can LLM agents autonomously discover and validate scaling laws?

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SLDAgent is an evolution-based LLM agent that automatically discovers scaling laws superior to human-derived counterparts by co-evolving symbolic expression and parameter-optimization subroutines. The authors also introduce SLDBench, an 8-task benchmark built from over 5,000 real LLM training experiments. SLDAgent paired with GPT-5 achieves avg \(R^2 = 0.748\) across all tasks, beating both human experts (\(R^2 = 0.517\)) and all existing coding agents.

Problem

Scaling law discovery (SLD) requires proposing a symbolic, generalizable formula relating model/data variables (e.g., \(N\), \(D\), \(E\)) to performance (e.g., loss \(L\)), but the formula space is infinite and current practice relies on slow, case-specific human experimentation. Existing LLM agents fail to consistently outperform human-derived laws on this open-ended symbolic regression task.

Method

SLDBench formalizes SLD: given a dataset \(\mathcal{D} = \{(x_i, j_i, y_i)\}_{i=1}^m\) of observed experiments, produce a symbolic expression \(f_\theta : x \mapsto \hat{y}\) and per-setting parameters \(\{\theta_j\}_{j \in C}\) that extrapolate accurately to larger-scale, held-out \(x\) (evaluated by \(R^2\)).

SLDAgent is an evolutionary program-search agent that co-optimizes two subroutines: 1. Expression – implements the symbolic law \(f_\theta(x, \theta)\) 2. Optimization – finds \(\hat{\theta} = \arg\min_\theta \mathcal{L}(y, f_\theta(x))\) (e.g., BFGS)

At each evolution step, a parent program is sampled from a database using a 70/20/10 exploitation/diversity/elitism mixture. The LLM receives the parent code plus high-scoring "inspiration" programs and proposes a diff (modifying the law form, optimizer, or global variables). The child is executed, its training-set \(R^2\) is computed, and it is added to the database. The system uses a multi-island MAP-Elites strategy (5 islands) structured by score, complexity, and novelty to prevent premature convergence. The test set is never touched during evolution.

Key Contributions

  • SLDBench: First comprehensive, open-ended scaling law discovery benchmark; 8 tasks drawn from real LLM training data (parallel, vocab_size, sft, domain_mix, moe, d_constrain, lr&bsz, u_shape) totaling ~5,000+ experiments.
  • SLDAgent: Evolutionary coding agent achieving state-of-the-art, superhuman SLD performance that generalizes across LLM backends.
  • Principled law formulations: Qualitative analysis showing SLDAgent-discovered laws have improved dimensional consistency and asymptotic behavior vs. human laws (e.g., dimensionless ratio \((D/\theta_3)^{\theta_1}\) in the SFT law vs. additive \(D^{\theta_1} + \theta_3\)).
  • Practical validation: Discovered laws enable analytic pre-training hyperparameter optimization and fine-tuning checkpoint selection.

Results

  • SLDAgent (GPT-5): avg \(R^2 = \mathbf{0.748}\) vs. Human \(0.517\), Goose (GPT-5) \(0.695\), CodeX (GPT-5) \(0.550\) (Table 2).
  • SLDAgent improves over its native CLI baseline across every tested model family:
  • Gemini-2.5-Flash: \(0.077 \to 0.506\) (+0.429)
  • Gemini-3-Pro: \(0.382 \to 0.636\) (+0.254)
  • Claude-Haiku-4.5: \(0.419 \to 0.519\) (+0.100)
  • Claude-Sonnet-4.5: \(0.472 \to 0.590\) (+0.118)
  • o4-mini (CodeX): \(0.349 \to 0.657\) (+0.308)
  • GPT-5 (CodeX): \(0.550 \to 0.748\) (+0.198)
  • SLDAgent (GPT-5) matches or exceeds the human baseline on every individual task.
  • lr&bsz and u_shape remain hardest; many agents return negative \(R^2\) on these tasks, and the human reference scores \(-0.076\) and \(-1.000\) respectively.

Limitations

  • u_shape remains the primary unsolved outlier: SLDAgent (GPT-5) achieves only \(R^2 = -0.305\), still negative.
  • Weaker backbone models (e.g., Gemini-2.5-Flash, Claude-Haiku-4.5) still fall below the human reference on the hardest tasks (lr&bsz), revealing model-capacity bottlenecks.
  • SLDBench covers only 8 tasks; the authors note plans to expand but coverage is currently limited to scenarios with open-source data.
  • The evolutionary loop requires a fixed compute budget of iterations, and the framework inherits the costs of running LLM inference per evolution step at scale.

Relevance to Agentic AI / LLM Agents

This paper is a direct demonstration that evolutionary program synthesis — where an LLM iteratively mutates executable code and is guided by a verifiable objective — can surpass human experts on a hard, open-ended scientific discovery task. It advances the "AI scientist" agenda by showing agents need not generate hypotheses from scratch each iteration: co-evolving both the symbolic model and its fitting routine unlocks search over formula spaces unreachable by single-pass coding agents. The SLDBench benchmark provides a clean, gradient-free, reward-model-free evaluation surface (raw extrapolation \(R^2\)) that is a strong testbed for future agentic reasoning and scientific discovery systems.