Do Large Language Models Know What They Are Capable Of?

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Evaluates LLM self-assessment of task success; calibration crucial for autonomous agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper empirically investigates whether LLMs can accurately predict in-advance whether they will succeed on a given task, across three experiment types: single-step coding tasks, a sequential resource-acquisition scenario with costly failure, and multi-step agentic SWE-Bench tasks. All tested LLMs are systematically overconfident, yet most retain better-than-random discriminatory power. Critically, larger/newer models do not reliably improve calibration, and reasoning LLMs offer no advantage over non-reasoning models on self-assessment accuracy.

Problem

Prior work on LLM confidence estimation focuses almost exclusively on after-the-fact calibration (confidence stated after generating an answer) and single-step tasks. The gap is in-advance confidence — whether a model can predict success before attempting a task — and how that estimate evolves during multi-step agentic execution. This matters for deployment safety: an agent that cannot predict its own failure will attempt high-stakes actions it is likely to fail, incurring real costs or misuse risks.

Method

Three experiments are conducted across Llama, GPT, and Claude model families:

1. Experiment 1 (single-step): For each of 1,140 BigCodeBench Python tasks, the LLM is separately prompted for an in-advance success probability \(\hat{p}_i\) and then prompted to solve the task. Discriminatory power is measured via AUROC on the \((\hat{p}_i, \text{outcome}_i)\) pairs.

2. Experiment 2 (in-context learning under costly failure): The LLM is presented with 512 sequences of 9 work contracts sequentially; each contract is a BigCodeBench task with a symmetric \(\pm\$1\) reward/penalty. The LLM states \(\hat{p}\) and decides accept/decline before each contract. Prior contracts, submissions, and outcomes remain in-context. Performance is tracked as AUROC, predicted success rate, acceptance rate, and expected profit across contract positions \(n=1,\ldots,9\).

3. Experiment 3 (multi-step agentic): LLMs solve SWE-Bench Verified tasks (499 tasks, 70-tool-call budget). After each tool call \(s\), the LLM is prompted for \(\hat{p}_{i,s}\); an after-the-fact estimate is collected post-submission. AUROC at each step \(s\) is compared to step-1 AUROC to measure within-task calibration drift.

LLM rationality is evaluated by fitting expected-utility functions to the contract accept/decline decisions in Experiment 2 and testing consistency with expected-utility maximization.

Key Contributions

• Demonstrates that all tested LLMs are overconfident but mostly discriminative (AUROC > 0.5) on in-advance coding-task predictions.
• Shows that larger/newer LLMs within Llama and GPT families do not improve in-advance calibration; Claude models are the exception, showing a monotonically decreasing overconfidence trend.
• Establishes that several frontier LLMs (Claude Sonnet models, GPT 4.5) learn to reduce overconfidence and improve profitability through in-context failure experience, primarily via risk aversion rather than improved discrimination.
• Finds that reasoning LLMs (o1, GPT 5.1, Sonnet 3.7/4.5) do not outperform non-reasoning LLMs on self-assessment accuracy or calibration during multi-step tasks.
• Shows that on SWE-Bench tasks, Claude models' AUROC degrades in later task steps while most OpenAI models improve, and that after-the-fact reflection substantially recovers AUROC for Sonnet models.
• Demonstrates that LLMs' decisions are approximately rational given their (miscalibrated) probability estimates — poor decisions stem from wrong beliefs, not irrational choice mechanics.

Results

• Overconfidence (Exp. 1): All tested LLMs overestimate their success rate on BigCodeBench; Claude models show a monotonically decreasing overconfidence trend across model generations; Llama and GPT show no such trend.
• AUROC (Exp. 1): Most LLMs exceed the random baseline (AUROC = 0.5); Claude models have lower absolute AUROC than several Llama/GPT models but are the only family showing an improving trend.
• In-context learning (Exp. 2): Claude Sonnet 3.5 achieves near-perfect contract acceptance rate (~50%) by contract 5 (from a much higher baseline), substantially raising expected profit, while GPT 4.1 shows almost no change in acceptance rate or profit.
• Expected profit (Exp. 2): Claude Sonnet models and GPT 4.5 exhibit substantially increased expected profit by contract 9; improvement is driven by acceptance-rate reduction (risk aversion), not AUROC gain.
• Rationality (Exp. 2): All LLMs' accept/decline decisions are consistent with expected-utility maximization given their stated \(\hat{p}\) — irrational choice mechanics are not the bottleneck.
• Multi-step calibration (Exp. 3): All Claude Sonnet models become more overconfident as tasks progress; only GPT 4o substantially reduces overconfidence mid-task. Only 4 of 10 LLMs (GPT 4o, 4.1, 5.1(none), o1) show significantly higher after-the-fact AUROC than step-1 AUROC.
• Reasoning LLMs (Exp. 3): Reasoning models perform comparably to or worse than non-reasoning counterparts on both overconfidence and AUROC across multi-step tasks.

Limitations

• Coding tasks (BigCodeBench, SWE-Bench) may not generalize to other agentic domains (tool use, planning, open-ended reasoning).
• Experiment 2 uses a symmetric \(\pm\$1\) payoff; real-world risk/reward asymmetries could shift LLM behavior.
• Reasoning token budget was forced to 0 for single-step experiments to obtain in-advance estimates, which may suppress reasoning models' calibration capabilities in ways that don't reflect deployment.
• Only three model families tested; Apple, Gemini, Mistral, and others are absent.
• Results are cross-sectional snapshots; calibration may improve rapidly with future training, as authors note.
• The resource acquisition scenario is somewhat artificial; LLMs may behave differently when operating as fully autonomous agents with real tool access.

Relevance to Agentic AI / LLM Agents

Self-awareness of capability is a fundamental prerequisite for reliable autonomous agents: an agent that cannot predict when it will fail cannot make principled decisions to abort, escalate, or avoid high-stakes actions. This work provides the first systematic multi-condition evaluation of in-advance confidence on agentic tasks (SWE-Bench), directly quantifying how current frontier agents are miscalibrated during live execution. The finding that overconfidence often worsens during multi-step task progress — exactly when agents are already committed to an action trajectory — is a concrete safety-relevant failure mode for deployed LLM agents. The result that reasoning models do not improve self-assessment challenges the assumption that chain-of-thought training yields better introspection, and has direct implications for agent frameworks that rely on self-monitoring or reflective stopping criteria.