Doomed from the Start: Early Abort of LLM Agent Episodes via a Recall-Controlled Probe Cascade¶

🕒 Published (v1): 2026-07-07 17:03 UTC · Source: Arxiv · link

Why this paper was selected

Probe cascade detects doomed trajectories early; direct compute savings in deployed agentic systems

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

LLM agent episodes that are destined to fail can be detected early from internal hidden-state activations, before behavioral signals become informative. The paper proposes a recall-controlled cascade of per-round abort gates whose thresholds are calibrated with Clopper–Pearson confidence bounds, jointly optimized to maximize compute savings under a user-specified episode-level success recall floor. At a 90% recall target, the cascade saves up to 47.1% of inference compute—1.6–1.7× better than any single-gate policy.

Problem¶

Multi-step LLM agent episodes frequently enter unrecoverable failure states within the first few rounds but continue consuming large amounts of inference compute until the episode times out. Existing behavioral monitors become informative only at rounds 3–4, by which point much of the recoverable compute is already spent. Moreover, no prior abort policy provides a controllable, formally certified guarantee on the rate at which successful episodes are mistakenly aborted, and sequential multi-gate policies compound per-round error rates in a way that individual per-gate guarantees do not address.

Method¶

The method trains lightweight per-round linear probes (logistic regression, L2 regularization $C=1$) on the residual-stream hidden state at the final action token of round $r$, extracted from a single layer per model (layer 14/28 for Llama-3.2-3B; layer 20/28 for Qwen-2.5-7B), via a teacher-forced forward pass over the trajectory. These probes predict eventual episode failure $1-y$.

Each probe feeds a calibrated gate: for a per-round recall budget $t_r$, let $S_r$ be the set of failure scores of successful calibration episodes alive at round $r$, and let $k(\tau)$ count those with score $\leq \tau$. The threshold $\tau_r$ is the smallest value satisfying $$p(\tau_r) = \mathrm{Beta}^{-1}(\alpha;\, k(\tau_r),\, n_r - k(\tau_r) + 1) \geq t_r,$$ so the gate provably passes at least a $t_r$ fraction of successes with confidence $1-\alpha$.

The budget vector $\mathbf{t} = (t_1, \ldots, t_{R_g})$ (with $R_g = 6$) is selected by grid search ($t_r \in \{0.85, 0.90, 0.95, 0.98, 0.99, 1.0\}$; $6^6 = 46{,}656$ candidates) on a held-out validation split, maximizing token savings subject to either (a) a heuristic safety margin $\hat{\rho}_\text{val} \geq \rho^\star + \delta$ ($\delta=0.02$), or (b) a formal distribution-free certificate: the Clopper–Pearson lower bound on global recall exceeds $\rho^\star$. Tasks are partitioned 20/20/60 (calibration/validation/test) with group-stratified k-fold cross-fitting so no task's episodes appear on both sides of any split. The data requirement for the certified variant is explicit: certifiable targets are capped near $\rho^\star \lesssim \alpha_m^{1/n_\text{pos}}$ (the "rule of three").

Key Contributions¶

Demonstrates that eventual episode failure is linearly decodable from hidden activations as early as round 1—before behavioral evidence accumulates—with probe AUC 0.12–0.21 above the behavior-only scorer at rounds 1–2.
Proposes the first abort policy for LLM agents with a provable episode-level success recall guarantee across multiple sequential gates, using distribution-free Clopper–Pearson calibration and optional global certificate.
Shows that jointly searching the per-round recall budget allocation—rather than concentrating it at a single gate—yields 1.6–2.3× more compute savings than the best single-gate policy at every tested recall target.
Characterizes the sample complexity of certification: $n_\text{pos} \geq \ln \alpha_m / \ln \rho^\star$ successful validation episodes are required to certify a given $\rho^\star$, making unattainable promises visible before deployment.
Demonstrates that stacking observable behavioral features onto hidden-state probes provides no additional gain, confirming the internal signal subsumes the behavioral one.

Results¶

Qwen-2.5-7B cascade: $47.1\% \pm 10.3\%$ compute saved at $\rho^\star = 0.90$; $32.0\% \pm 12.2\%$ at $\rho^\star = 0.95$; vs. single-gate $27.6\% \pm 5.6\%$ and $17.4\% \pm 5.5\%$ respectively (1.7–1.8× improvement).
Llama-3.2-3B cascade: $37.2\% \pm 8.8\%$ at $\rho^\star = 0.90$; $19.9\% \pm 9.2\%$ at $\rho^\star = 0.95$; vs. single-gate $23.6\% \pm 5.3\%$ and $10.4\% \pm 3.9\%$ (1.6–2.3× improvement depending on target).
Achieved global recall matches or exceeds target in 7 of 8 agent–target configurations; worst deviation is $0.001$ below target (within seed noise).
Behavior-only cascade at $\rho^\star = 0.95$: Qwen saves $\sim 17.0\%$, roughly half the probe cascade's $32.0\%$; stacking surface features adds nothing ($\leq$ noise).
Uniform budget allocation collapses near zero savings at $\rho^\star \geq 0.95$, confirming that budget placement across rounds drives savings.
Certified variant: with $n_\text{pos} = 113$–$115$ episodes, targets $\geq 0.98$ are unattainable regardless of scorer quality; threshold is $\approx 0.974$.
Probe AUC advantage over surface scorer at round 2: 0.12–0.21 absolute, evaluated on TextCraft over 800 episodes per agent.

Limitations¶

Evaluated on a single environment (TextCraft from AgentGym) with binary task success; generalizability to richer, continuous-reward domains is unverified.
Requires white-box access to hidden activations, which is unavailable for API-only deployments.
Teacher-forced re-encoding of the trajectory at each gate round adds overhead not counted in the reported savings.
Probe training requires a labeled episode dataset; the certification data requirements ($n_\text{pos} \geq 113$ for $\rho^\star \leq 0.974$) may be prohibitive in low-data settings or for very high recall promises.
The guarantee is distribution-free but relies on the calibration and validation splits being exchangeable with test; distribution shift between deployment and collection can invalidate it.
Only logistic regression probes are studied; richer nonlinear classifiers could extract more signal but are not explored.

Relevance to Agentic AI / LLM Agents¶

This work directly addresses compute efficiency and reliability in deployed LLM agent systems—two central concerns as agents scale to longer-horizon tasks. The finding that failure is linearly decodable from internal representations in round 1 implies that modern LLMs encode task-difficulty and commitment very early, a mechanistic insight with implications for agent design, chain-of-thought analysis, and self-monitoring. The cascade framework is practically deployable and connects to the broader literature on agent monitoring, early-exit inference, and conformal prediction applied to sequential decision making. For researchers studying agent robustness and resource-efficient multi-step inference, the explicit sample-complexity characterization of certified recall targets provides a principled framework for trading off safety guarantees against data requirements.