Measuring AI Ability to Complete Long Software Tasks

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

METR introduces the 50%-task-completion time horizon — the human time to complete tasks that an AI agent solves with 50% success — as a unified, cross-benchmark metric for AI capability. Evaluating 12 frontier models (2019–2025) on 170 software/ML tasks, they find this horizon has doubled roughly every 7 months, reaching ~110 minutes for o3. Extrapolating the trend predicts AI capable of autonomous 1-month software tasks (167 work-hours) between mid-2028 and mid-2030.

Problem

Existing AI benchmarks saturate quickly and lack a common, intuitive scale that enables comparison across models of vastly different capability (e.g., GPT-2 vs. Claude 3.7). Benchmark scores also provide no grounded sense of real-world economic significance, making it hard to forecast when AI will automate substantive human labor.

Method

Three task suites are combined into 170 tasks: HCAST (97 diverse software tasks, 1 min–30 hrs), RE-Bench (7 ML research engineering tasks, each 8 hrs), and the new SWAA (66 atomic single-step tasks, 1–30 s). Skilled human baseliners (≥5 yrs experience) completed most tasks, recording wall-clock time; over 800 baselines totaling 2,529 hours were collected. For each AI agent, per-task binary success rates are computed over 8 runs, then a logistic regression is fit:

p_success(agent, task) = σ((log h_agent − log t_task) · β_agent)

where t_task is the geometric mean human completion time and h_agent is the learned 50% time horizon. Log(time horizon) is then regressed linearly against model release date to extract the exponential doubling trend.

Key Contributions

• Defines the 50%-task-completion time horizon as a human-referenced, benchmark-agnostic capability metric.
• Introduces SWAA (Software Atomic Actions): 66 short (1–30 s) tasks providing coverage of pre-2023 models.
• Documents exponential growth: doubling time ≈207 days (95% CI: 166–240 days) from 2019–2025.
• Qualitative failure analysis distinguishing GPT-4 vs. o1 failure modes (premature abandonment vs. repeated failed actions).
• External validity experiments: SWE-bench Verified replication, "messiness" scoring of tasks, and internal PR evaluation with contractor vs. maintainer baselines.

Results

• GPT-2 (2019): 50% time horizon of ~2 seconds; o3 (2025): ~110 minutes.
• Doubling time for 50% horizon: 207 days (≈7 months); 80% horizon doubling time: 204 days — nearly identical, but 80% horizons are 4–6× shorter.
• Success rate vs. log(human time): exponential decay, R² ≈ 0.80.
• SWE-bench Verified replication confirms exponential trend but with a shorter doubling time (~70 days), attributed to differential underestimation of easier tasks' human time.
• o3 lies above the long-run trend line (p = 0.006), suggesting possible acceleration post-2024.
• Extrapolation: 80% CI for 1-month-horizon AI spans mid-2028 to mid-2030; as early as early 2027 if 2024–2025 acceleration continues.
• Cross-domain preliminary analysis (math competition, video QA, autonomous driving) finds similar doubling times.

Limitations

• Tasks are cleaner and more structured than real-world software work; models perform measurably worse on higher-"messiness" tasks (lacking clear feedback loops or requiring proactive information gathering).
• No tasks with very long horizons (>8 hrs) exist in sufficient quantity to measure 95%+ success rates reliably.
• External validity unconfirmed: real tasks rarely have automatic scoring and often involve dynamic environments, resource limits, and multi-agent coordination.
• Human baseline times vary enormously by reference population (contractors took 5–18× longer than maintainers on internal PRs), creating metric sensitivity to baseliner choice.
• Only 7 frontier models in the 2024–2025 window, giving low confidence in the accelerated recent slope.
• Trend extrapolation ignores potential structural breaks from agency training, compute saturation, or recursive AI R&D automation.

Relevance to Agentic AI / LLM Agents

This paper provides the most rigorous longitudinal capability measurement to date for long-horizon autonomous agents, directly quantifying how far agents can execute without human intervention. The doubling-every-7-months finding contextualizes competing agentic benchmarks (SWE-bench, TheAgentCompany, RE-Bench) on a single scale, which is essential for understanding where current scaffolding and tool-use improvements actually move the needle. The failure taxonomy (poor planning, premature abandonment, repetition of failed actions) pinpoints specific agentic control-flow deficiencies that guide scaffold and training design. The 1-month horizon threshold is also an operationalized definition of economically transformative agency, relevant both to capability forecasting and AI safety risk modeling.