Skip to content

Scaling Laws for Economic Productivity: Experimental Evidence in LLM-Assisted Consulting, Data Analyst, and Management Tasks

🕒 Published (v1): 2025-12-24 18:24 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

A preregistered RCT with 500+ professionals across 13 LLMs derives empirical "Economic Scaling Laws" mapping training compute and calendar-time model progress to measurable productivity gains. Each year of frontier model progress reduces professional task completion time by ~8%, with 56% of gains from compute scaling and 44% from algorithmic progress. Agentic workflows (multi-step tool use) benefit far less than non-agentic analytical tasks, and human users actively suppress quality gains from frontier models.

Problem

ML scaling laws relate compute to cross-entropy loss, but the elasticity of real-world professional productivity with respect to model capability is unquantified. Prior RCTs study single domains or treat AI as a binary; no study measures how productivity gains change as a function of model compute or release date across a representative multi-profession sample.

Method

Preregistered RCT (AEA registry AEARCTR-0013743): 500+ professionals (consultants, data analysts, managers) recruited via Prolific with ~90% screen-out rate; 79% had 3+ years of experience. Participants randomly assigned to control or one of 13 LLMs spanning ~4 orders-of-magnitude compute range and ~3 years of release dates (Nov 2022–2025). Tasks classified as agentic (multi-step tool use: Gantt charts, PDF extraction + stats) or non-agentic (analytical/writing: A/B test interpretation, report writing). Performance incentivized: $15 base + $15 bonus for grades ≥5/7 by expert peer graders. Productivity measured as Earnings Per Minute (EPM) and Total Earnings Per Minute (TEPM). Scaling laws estimated by regressing log time/EPM/TEPM on (1) months since Nov 2022 and (2) log training compute; decomposition combines both regressors to separate compute vs. algorithmic contributions.

Key Contributions

  • First empirical derivation of Economic Scaling Laws: quantified elasticity of professional output w.r.t. training compute (−6.3% task time per 10× compute) and calendar time (−8%/year).
  • Decomposition of AI progress: 56% compute scaling, 44% algorithmic progress.
  • Agentic vs. non-agentic heterogeneity: TEPM gains of $1.58/min (non-agentic) vs. $0.34/min (agentic), significant at p=0.043.
  • "Last-mile" quality finding: autonomous model output quality scales with compute (p<0.01, +0.51 grade points per 10× compute), but human-assisted output quality is flat across all models (p≈0.85), with frontier model outputs actively degraded by human intervention.
  • Aggregate projection: ~20% U.S. productivity growth over the next decade using Acemoglu (2024) framework with updated elasticities.

Results

  • Any AI vs. control: EPM +81.3% (\(0.56/min, p=0.001); quality +0.34 SD (p<0.001); TEPM +146% (\)1.06/min, p<0.001).
  • Calendar-time scaling: −8%/year task time (p≈0.04); +\(13.97/hour EPM per year; +\)26.30/hour TEPM per year.
  • Compute scaling: −6.3% task time per 10× compute (p≈0.31, less precise).
  • Non-agentic: −10.7%/year time reduction (p<0.05); +$20.40/hour EPM per year.
  • Agentic: −4.8%/year time reduction (not significant).
  • AI-only quality: +0.51 grade points per 10× compute (p<0.01); frontier models reach >6/7 vs. 3.52 human baseline.
  • Human+AI quality: flat ~4.35/7 across all compute levels (p≈0.85).
  • Aggregate: estimated 19.9% U.S. GDP productivity gain over 10 years.

Limitations

  • Tasks are short-horizon (20–60 min); not representative of multi-day corporate projects.
  • Participants used text-only chatbot interfaces with limited tool access, likely underestimating true agentic capabilities of current models.
  • Compute scaling coefficient estimated imprecisely (p≈0.31); 56/44 decomposition carries uncertainty.
  • No general equilibrium effects (wages, employment, task-structure shifts).
  • Extrapolation assumes current transformer scaling paradigm continues and does not model architectural discontinuities.
  • Individual-level productivity only; no team or organizational dynamics.

Relevance to Harnesses / Meta-Harnesses

The agentic vs. non-agentic performance gap (~4.6× TEPM advantage for non-agentic tasks) is direct empirical evidence that harness design—how tools are surfaced, sequenced, and mediated to LLMs—is a first-order productivity variable, not an implementation detail. The finding that human users suppress frontier model quality in the "last mile" implies that meta-harnesses which reduce or restructure human-in-the-loop intervention (e.g., automated quality gates, agentic loops that bypass premature human editing) could unlock the scaling gains that current chatbot interfaces forfeit. The paper's decomposition also provides a quantitative baseline for evaluating whether a harness architecture narrows the agentic/non-agentic gap as models scale.