Skip to content

From Agent Failures to Text Policies: What Works and What Breaks

🕒 Published (v1): 2026-07-22 19:08 UTC · Source: Arxiv · link

Why this paper was selected

Empirical analysis of TextGrad applied to agents; identifies what breaks in text-gradient optimization for agent components

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper dissects why TextGrad-style text optimization fails when applied to frozen LLM agents. Human-written policies yield a +5.0 success-point gain on TextWorldExpress, proving useful policy text exists and agents can follow it, but policies learned autonomously from trajectories do not reliably outperform fixed prompting even with richer traces, counterfactual branches, or multi-round GEPA search. The bottleneck is generating and selecting good rules from experience, not executing them.

Problem

Frozen LLM agents (API-only, too costly to fine-tune) are appealing candidates for text-based optimization, but agent trajectories create a credit-assignment problem: a final failure does not reveal which specific action in the sequence was wrong or what reusable lesson should be extracted. Existing work (TextGrad, ProTeGi, GEPA) does not cleanly separate the capacity to follow a policy from the ability to learn one, leaving it unclear where the optimization pipeline actually breaks.

Method

The authors introduce RulePI, a prototype that separates the pipeline into four testable components: (1) execution — can the frozen agent follow a useful rule?; (2) evidence — does the trajectory expose the decisive decision?; (3) update — can the learner generate a reusable rule from that evidence?; (4) selection — can development validation keep helpful rules and discard harmful ones?

Formally, one update cycle is: $\(\tau \sim \pi_\theta(\cdot \mid x), \quad g_{\text{text}} = D(\tau, R(\tau)), \quad x' = U(x, g_{\text{text}})\)$ where \(x\) is the current text policy, \(\theta\) is frozen, \(D\) is a diagnosis function, and \(U\) is the updater. The rule \(x'\) is frozen before test and applied without any test-time trajectory.

Four conditions are tested on Qwen2.5-7B-Instruct and Mistral-7B-Instruct-v0.3 across TextWorld, TextWorldExpress, and TextArena: - Capacity: human-written rules + correct family routing (privileged upper-bound test) - Evidence: compact summary vs. context-bounded step-by-step trace - Counterfactual: same-state action branches to isolate credit - Search: official GEPA with multi-round mutation and Pareto selection

Evaluation uses 95% cluster-bootstrap intervals (10,000 resamples) with separate training, development, and held-out test seeds.

Key Contributions

  • Decomposes agent-level TextGrad into four independently testable failure modes (execution, evidence, update, selection), providing a diagnostic framework rather than another optimizer.
  • Demonstrates a clear capacity–learning gap: human policies lift pooled success by +5.0 pp (CI: +1.88, +9.38), but no learned condition reliably beats fixed prompting end-to-end.
  • Shows that richer evidence (bounded traces vs. summaries) improves intermediate comparisons by +3.75 pp but does not translate to held-out gains.
  • Shows counterfactual branches marginally improve credit (+0.31 pp over retry) without yielding end-to-end improvement.
  • Shows GEPA search (+1.25 pp, CI spanning zero) with model-level disagreement (Qwen −1.25 pp, Mistral +3.75 pp).
  • Provides a manual taxonomy of 32 generated rules: only 7/32 are grounded executable strategies; 9 are too generic; 16 are shortcuts, semantically wrong, or malformed.
  • Identifies two distinct failure transitions: trajectory → scoped policy (update failure) and candidate policy → deployment decision (selection failure).

Results

  • Capacity (human policy): pooled success 15.63% → 20.63% (+5.00 pp, CI: +1.88, +9.38); mean interaction reduced by 2.64 turns (CI: −5.15, −0.41).
  • Evidence (trace vs. summary): +3.75 pp (CI: 0.00, +8.12) favoring traces; trace vs. fixed: −1.88 pp (CI: −5.62, +1.88) — below fixed prompting.
  • Counterfactual (branch vs. retry): +0.31 pp (CI: 0.00, +0.94); branch vs. fixed: −1.25 pp (CI: −3.75, 0.00).
  • GEPA vs. fixed: +1.25 pp (CI: −3.13, +5.63), non-significant; Qwen −1.25 pp, Mistral +3.75 pp (model-level disagreement).
  • Rule taxonomy: 7 grounded, 9 generic heuristic, 5 instance shortcuts, 9 semantic errors, 2 malformed (of 32 rules).
  • Structured actor control (appendix only): stored policies use 46.5% fewer actions than retry+persistence on TextWorld and 41.0% fewer on TextArena at matched success.

Limitations

  • Evidence covers only two 7B models (Qwen2.5-7B, Mistral-7B) and three text-game benchmarks; does not generalize to web, software, robotics, or safety-critical domains.
  • Human-policy condition is a privileged upper bound (rules and family routing are hand-supplied), not an autonomous method.
  • Trace study shortens 22% of observations to fit context; this measures richer-but-bounded evidence, not truly unbounded traces.
  • Counterfactual study yields only 7 valid grounded rules — too few for strong conclusions about the update operator.
  • Development validation sets contain only 8 instances per family, limiting selection reliability.
  • GEPA test uses the same 7B model for acting and reflection, uses more rollouts than single-rule studies, and only covers one benchmark.
  • No matched comparisons against Reflexion, ExpeL, RAPOA, or weight-training methods.
  • All intervals are exploratory and unadjusted for multiple comparisons; monetary, latency, energy, and token costs are not reported.
  • Rule taxonomy is single-coder, not blinded, and not independently double-coded.

Relevance to Agentic AI / LLM Agents

This paper provides a principled failure analysis for a central challenge in LLM agent optimization: how to improve frozen agents through textual policy learning. By cleanly separating execution capacity from learning ability, it quantifies the gap that practitioners face when applying prompt optimization to sequential decision-making — a gap that is invisible when only end-to-end benchmarks are reported. The finding that the bottleneck lies in rule generation and selection (not rule following) directly informs the design of future agent-level optimizers such as GEPA, RAPOA, and Feedback Descent, suggesting that investment should go into structured candidate generation, family-aware selection, and calibrated validation rather than into more rollouts alone. The diagnostic decomposition (execution → evidence → update → selection) is a reusable evaluation protocol for any trajectory-to-policy pipeline.