CodeRescue: Budget-Calibrated Recovery Routing for Coding Agents¶

🕒 Published (v1): 2026-07-21 17:56 UTC · Source: Arxiv · link

Why this paper was selected

Budget-calibrated failure-recovery routing for coding agents; practical cost-aware scaffolding mechanism

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

CodeRescue frames post-failure recovery in coding agents as a three-way routing problem—reflect, replan, or escalate—and trains a supervised router over execution rollouts. A Conformal Risk Control (CRC) layer then calibrates a deployment-time cost penalty to hit user-specified budget targets without retraining. The CRC-calibrated system exceeds always-escalate solve rate while using 35% of its mean recovery cost.

Problem¶

Existing cost-aware LLM cascades treat a cheap model's failure primarily as evidence that a stronger model is needed, ignoring that execution feedback (compiler errors, failed assertions, stderr traces) can turn a coding failure into a tractable cheap-model repair problem. No prior work decides, in a budgeted way, which recovery action to take after failure—repair with feedback, replan from scratch, or escalate—especially when success patterns across actions are non-monotone and cost–quality budgets vary across deployments.

Method¶

Recovery context $x = (q, v_0, e_0)$: problem statement, execution verdict, and stderr trace. Action set $\mathcal{A} = \{\text{reflect, replan, escalate}\}$. A supervised router (Qwen3.5-4B, full-parameter fine-tuned) is trained on execution rollouts labeled with the cheapest successful action $a^\dagger(x) \in \arg\min_{a \in S(x)} c(a,x)$.

Cost-regularized policy: $$\pi_\lambda(x) = \arg\max_{a \in \mathcal{A}} \{s_\theta(a \mid x) - \lambda c(a, x)\}$$

where $\lambda \geq 0$ is a scalar cost penalty. For fixed $x$, the selected action switches between actions $a$ and $b$ at breakpoint $\lambda_{ab}(x) = \frac{s_\theta(a|x)-s_\theta(b|x)}{c(a,x)-c(b,x)}$. The union of all pairwise breakpoints over the calibration set yields a finite sorted grid $\Lambda_K$.

CRC calibration: Given budget $B$, the deployed penalty is $$\hat{\lambda} = \min\!\left\{\lambda_k \in \Lambda_K : \frac{n\,\hat{C}_n(\lambda_k) + c_{\max}}{n+1} \leq B\right\}$$ where $c_{\max}$ is the cost cap and the additive correction implements a leave-one-out conformal bound. Under exchangeability, this guarantees $\mathbb{E}[c(\pi_{\hat{\lambda}}(X_{n+1}), X_{n+1})] \leq B$ marginally over calibration and test. Adapting to a new budget requires only re-scanning the pre-computed cost table—no retraining.

Key Contributions¶

Post-failure recovery routing: formalizes the three-action recovery decision over heterogeneous actions with non-monotone success patterns.
Budget-controllable deployment via CRC: one trained router yields multiple budgeted operating points; the mean-cost guarantee holds marginally without per-budget retraining.
Empirical characterization of recovery heterogeneity: 28% of failures are cheap-only solvable, 45% require escalation, and 27% are solvable by both—showing the three actions are complementary, not ordered.

Results¶

Learned router (Qwen3.5-4B FT): 81.7% solve rate at 5.51 m$ mean cost on GPT-5.4-nano holdout (n=360 test).
Always-escalate baseline: 68.6% solve rate at 7.22 m$—the router exceeds it by +13.1 pp at lower cost.
Always-replan (best fixed cheap policy): 45.3% solve rate at 1.59 m$.
Prompt-only zero-shot routers: Claude Sonnet 4.6 achieves 45.3%, Gemini 3.1 Pro 36.7%, GPT-5.4-nano 33.1%, GPT-5.4 32.8%—all far below the fine-tuned router.
CRC frontier: one calibrated operating point exceeds always-escalate solve rate while using ~35% of always-escalate mean recovery cost.
Oracle routing cost vs. always-escalate: mean oracle cost is substantially lower than always-escalate across all five benchmarks (e.g., 1.8 m$ vs. 3.7 m$ on TACO-medium, 0.7 m$ vs. 3.2 m$ on BigCodeBench).

Limitations¶

The three-action set (reflect, replan, escalate) is explicitly a minimal first-order set, not an exhaustive taxonomy of agent interventions.
CRC provides a marginal expected-cost guarantee, not a solve-rate guarantee; any solve-rate improvement is purely empirical.
Guarantee requires exchangeability of calibration and test invocations—violated if the failure distribution shifts across deployments.
Router trained on GPT-5.4-nano failures; transferability to other cheap models is not evaluated.
Roughly 30% of all attempted problems fail all three recovery actions and are excluded from router training, leaving a hard residual unsolved.
Evaluated only on algorithmic code-generation benchmarks; SWE-bench-style repository-level tasks are not covered.

Relevance to Agentic AI / LLM Agents¶

This work directly addresses the multi-step decision structure of coding agents: rather than treating a failed tool invocation as a terminal signal, it treats execution feedback as a first-class input for adaptive compute allocation. The CRC calibration layer is practically significant for agent deployment because it decouples training from budget policy, allowing a single router to serve diverse operating constraints at inference time. The empirical finding that cheap recovery and escalation are complementary—not ordered—has direct implications for how multi-model agentic systems should be designed, challenging the standard cascade assumption. The formalization of recovery routing connects test-time compute scaling literature with cost-aware orchestration, bridging two active research threads in agentic AI.