Failure as a Process: An Anatomy of CLI Coding Agent Trajectories

🕒 Published (v1): 2026-07-10 15:25 UTC · Source: Arxiv · link

Why this paper was selected

Earl T. Barr (UCL); empirical anatomy of coding-agent failure modes — directly actionable for harness builders

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper presents the first large-scale empirical study treating CLI coding-agent failure as a temporal process rather than a binary outcome. Analyzing 1,794 annotated trajectories (63,000+ steps) across 7 frontier models and 3 scaffolds, the authors find that decisive errors occur at a median of step 7, lock in by step 12, yet only become observable (if at all) around step 16 — meaning 28% of failures never surface. The dominant root cause is epistemic error (57.9%), not capability limitation.

Problem

Existing empirical studies of coding agent failure treat it as a static outcome — cataloguing what went wrong after the fact — rather than analyzing when failure becomes inevitable and how it silently propagates. Additionally, prior work focuses on issue-resolution or multi-agent settings, not the CLI/terminal environments where products like Claude Code and Codex actually operate. This leaves open questions about when to intervene, why errors form, and whether patterns generalize across model/scaffold combinations.

Method

The authors collect 3,843 execution trajectories from 7 frontier models (Claude Sonnet 4, GPT-5, Gemini 2.5 Pro, Qwen3-480B, DeepSeek V3.2, Kimi K2, Devstral 2) across 3 scaffolds (MiniSWE, OpenHands, Terminus2) on Terminal-Bench (240 CLI tasks in containerized Docker environments). After filtering to the 89 tasks with complete coverage across all 21 model×scaffold systems, 1,794 valid trajectories (610 success, 1,184 failure) are retained.

Each failed trajectory is annotated with three timestamps: - \(t_\text{err}\): the decisive error step (error that ultimately determines failure, labeled retrospectively) - \(t_\text{lock}\): failure lock-in (first step after which no recovery is empirically observed) - \(t_\text{obs}\): first observable failure signal (if any)

These define two derived intervals: fix window \(= t_\text{lock} - t_\text{err}\) and observability lag \(= t_\text{obs} - t_\text{lock}\).

Annotation uses an LLM-assisted pipeline: Claude Opus 4.6 generates structured drafts; two independent human annotators finalize all labels. Inter-annotator Cohen's \(\kappa\) = 0.78–0.94. Root causes are classified into a 9-type taxonomy under three categories (epistemic, competence, environment). A prefix monitor (Claude Sonnet 4.6 reading only the first \(t\) steps, blind to outcome) is evaluated on 2,659 prefixes from 600 trajectories to test real-time detectability.

Key Contributions

• Process-oriented framework decomposing failure into three stages (\(t_\text{err}\), \(t_\text{lock}\), \(t_\text{obs}\)) with derived measures for fix window and observability lag.
• 14 empirical findings across failure occurrence, root causes, recovery behavior, and cross-system consistency, drawn from the largest annotated CLI-agent trajectory dataset to date.
• Root-cause taxonomy of 9 error types grouped as epistemic, competence, and environment errors, applied at scale across 7 models and 3 scaffolds.
• Prefix monitor evaluation characterizing real-time detectability and the role of task specification in lifting recall.
• Public dataset: 1,794 annotated trajectories and 63,000+ steps released openly.

Results

• Decisive errors occur early: median \(t_\text{err} = 7\) steps (mean 11.9); median trajectory length is 27 steps, so failure is typically determined in the first quarter of execution.
• Fix window is narrow: median fix window (\(t_\text{lock} - t_\text{err}\)) = 1 step; 60.9% of decisive errors leave at least 1 recovery step, 43.9% leave ≥3 steps.
• Observability lag is large: median \(t_\text{obs} \approx 16\) steps — approximately 10 steps after the decisive error.
• 28% of failures never surface: the \(t_\text{obs}\) curve plateaus at 72%, meaning 28% of failed trajectories produce no observable failure signal before termination.
• Epistemic errors dominate: 57.9% of decisive errors are epistemic (false premise 30.7%, specification neglect 14.9%, output misreading 4.4%, ignored signal 4.1%, premature action 3.7%); competence errors account for 32.8% (knowledge gap 24.0%, capability limitation 8.8%); environment blockers 8.8%.
• Real-time detection is poor: prefix monitor achieves 82% precision after lock-in but only 3.7–8.7% of failures are flagged before lock-in; recall rises from 18.2% to 28.8% when task requirements are provided to the monitor (false-positive rate on successful runs stays 2–3%).
• Specification-relative errors require goal exposure: recall on ignored-requirement errors improves from 3% → 22%, false-premise from 15% → 32%, and capability-gap from 12% → 30% when requirements are added to the monitor.

Limitations

• Terminal-Bench only: findings are scoped to 89 tasks on one CLI benchmark; generalizability to issue-resolution or multi-agent settings is not established.
• Empirical unrecoverability: \(t_\text{lock}\) is defined as the step after which no recovery was observed, not proven to be theoretically impossible — the definition is observer-relative.
• Snapshot in time: models were evaluated incrementally from May 2025; results reflect a fixed snapshot and may not hold as models improve.
• Annotation subjectivity: even with high \(\kappa\), decisive error labeling requires retrospective semantic judgment; the taxonomy may not be exhaustive.
• Prefix monitor is a single model: real-time detection experiments use one monitor (Claude Sonnet 4.6), leaving open whether stronger or specialized monitors could substantially close the detection gap.
• No causal intervention: the study is purely observational; it does not demonstrate that early intervention improves task success rates.

Relevance to Agentic AI / LLM Agents

This paper directly characterizes the internal failure dynamics of LLM-based CLI agents — the same class of systems as Claude Code, Codex CLI, and Gemini CLI — providing empirically grounded evidence that current agents commit decisive errors very early (median step 7) while producing no observable signal until much later. For researchers designing agent scaffolds, guardrails, or monitoring systems, the finding that 28% of failures are entirely silent and that specification-aware prefix monitoring only marginally improves early recall (3.7–8.7% pre-lock-in) motivates new directions: proactive verification at task onset, tighter integration of task requirements into agent self-monitoring, and error-type-aware intervention strategies. The root-cause taxonomy — particularly the dominance of epistemic over competence errors — suggests that the bottleneck for reliability is not raw model capability but information grounding and assumption verification during planning.