Failure as a Process: An Anatomy of CLI Coding Agent Trajectories¶

🕒 Published (v1): 2026-07-10 15:25 UTC · Source: Arxiv · link

Why this paper was selected

Empirical anatomy of CLI agent failure trajectories; actionable for harness designers improving reliability

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper presents the first large-scale empirical study of CLI coding-agent failure as a temporal process rather than a binary outcome, annotating 1,794 complete trajectories (1,184 failed, 610 successful) across seven frontier models and three scaffolds. The central finding is that decisive errors occur at a median of step 7, lock in by step 12, yet surface observable signals only at step 16—meaning failures are largely sealed before they are visible. Improving reliability requires earlier intervention in the trajectory, not post-hoc repair.

Problem¶

Existing empirical studies of coding-agent failure treat failure as a static label applied to a completed run. They do not model when failure becomes inevitable, how error chains evolve, or whether recovery is still possible at any given point. Additionally, most prior work targets issue-resolution or multi-agent settings, not terminal-based (CLI) agents, which are the native environment for products like Claude Code and Codex.

Method¶

The authors collect 3,843 execution trajectories from 3 scaffolds (MiniSWE, OpenHands, Terminus2) × 7 frontier models (Claude Sonnet 4, GPT-5, Gemini 2.5 Pro, Qwen3 480B, DeepSeek V3.2, Kimi K2, Devstral 2) on Terminal-Bench (240 containerized CLI tasks). After filtering to tasks with complete 21-combination coverage, 1,794 valid trajectories remain.

Three timestamps are annotated per failed trajectory: \(t_{err}\) (decisive error — the error that determines eventual failure, labeled retrospectively), \(t_{lock}\) (empirical failure lock-in — the last step from which recovery was ever observed), and \(t_{obs}\) (first externally observable failure signal). From these, two derived quantities capture recoverability: fix window \(= t_{lock} - t_{err}\) and observability lag \(= t_{obs} - t_{lock}\).

Annotation uses an LLM-assisted pipeline: Claude Opus 4.6 generates structured drafts with evidence citations; two independent human annotators then review the full trajectory and finalize every label. Inter-annotator agreement (Cohen's \(\kappa\)) ranges from 0.78 to 0.94. A root-cause taxonomy classifies each decisive error trigger into three categories: Epistemic, Competence, and Environment.

A prefix monitor (Claude Sonnet 4.6) is evaluated on 2,659 prefixes from 600 trajectories to test online failure detection, comparing two conditions: task name only vs. task name + core requirements.

Key Contributions¶

Process-oriented failure framework: decomposes failure into three stages (\(t_{err}\), \(t_{lock}\), \(t_{obs}\)) with derived metrics (fix window, observability lag).
Root-cause taxonomy: nine error types grouped into Epistemic (57.9%), Competence (32.8%), and Environment (9.4%) categories across 1,184 failed trajectories.
Largest CLI trajectory dataset: 1,794 annotated runs, 63,000+ execution steps, released publicly.
Prefix monitor experiment: quantifies the gap between retrospective and real-time failure detection, and the role of specification exposure in closing it.
Cross-system findings (RQ4): separates model-dependent from scaffold-dependent failure characteristics.

Results¶

Decisive errors occur early: median \(t_{err} = 7\) steps (mean 11.9), within the first quarter of a median 27-step failed trajectory.
Narrow fix window: median \(t_{lock} - t_{err} = 1\) step; 60.9% of decisive errors leave ≥1 recovery step, but the window is short.
Observability lag is large: median \(t_{obs} \approx 16\) steps (~10 steps after \(t_{err}\)); 28% of failures never produce an observable signal (silent failures).
Epistemic errors dominate root causes: 57.9% total — false premise (30.7%), specification neglect (14.9%), output misreading (4.4%), ignored signal (4.1%), premature action (3.7%).
Competence errors: knowledge gap (24.0%), capability limitation (8.8%).
Prefix monitor precision vs. recall gap: 82% precision at identifying already-locked-in failures, but median lead time relative to \(t_{lock}\) is 0; only 3.7–8.7% of failures are flagged before lock-in.
Specification exposure lifts recall: adding task requirements raises overall recall from 18.2% to 28.8%; ignored-signal errors jump from 3% to 22% recall; false-premise errors from 15% to 32%.
False-positive rate on successful early prefixes: 2–3%, confirming the monitor is not simply flagging hard tasks.

Limitations¶

Paper is truncated; only Findings 1–4 (of 14) and the root-cause taxonomy are visible; RQ3 (recovery behavior patterns) and RQ4 (cross-system differences) findings are absent from the provided text.
\(t_{lock}\) is empirically unrecoverable only — the study cannot prove theoretical irrecoverability, only that no recovery was observed in the data.
Trajectories were collected incrementally from May 2025; model versions are a fixed snapshot and results may not generalize to future model releases.
28.6% of trajectories were self-generated to fill missing scaffold/model combinations, introducing potential procedural heterogeneity with the 71.4% reused from prior work.
Filtering to 89 tasks (those with all 21 combinations complete) may select for systematically different task types than the full 240-task set.
The prefix monitor is a single model (Claude Sonnet 4.6); its detection limits may not generalize to other monitor architectures.

Relevance to Harnesses / Meta-Harnesses¶

This paper is directly relevant to harness design: the three scaffolds studied (OpenHands, MiniSWE, Terminus2) are themselves agent harnesses, and the study's finding that 28% of failures are silent—never emitting an observable error signal—argues that harnesses must embed proactive validity checks rather than relying on outcome signals alone. The prefix monitor experiment is a prototype meta-harness component: an online watchdog that reads trajectory prefixes and predicts lock-in, directly addressing the question of where a harness should inject intervention hooks. The finding that specification exposure (providing task requirements to the monitor) nearly doubles recall suggests that future meta-harnesses should pass structured task constraints as a first-class input to any monitoring layer. The three-timestamp decomposition (\(t_{err}\), \(t_{lock}\), \(t_{obs}\)) provides a formal vocabulary for reasoning about harness intervention points—particularly the fix window as the actionable window for harness-level repair or escalation.