Skip to content

ReflexGrad: Within-Episode Failure Recovery in LLM Agents via Progress-Gated Dual-Process Routing

🕒 Published (v1): 2025-11-18 15:25 UTC · Source: Arxiv · Venue: ICML 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ReflexGrad is a training-free, demo-free dual-process architecture for within-episode failure recovery in LLM agents. It routes between a fast TextGrad-style local refinement loop (every k=3 steps) and a slow Reflexion-style causal replanning pass (triggered by m=5 consecutive low-progress scores), merging updates under a deterministic priority rule. On ALFWorld 134 tasks, it lifts Qwen3-8B from 35.1% to 75.4% and GPT-5 from 46.3% to 88.1% without any demonstrations.

Problem

LLM agents frequently commit to a wrong strategy early, receive uninformative environment feedback, repeat minor variations, and exhaust their step budget. The recovery information exists in the post-failure trajectory, but no prior architecture exploits it within a single episode. Reflexion delays recovery to the next trial and requires demonstrations; TextGrad's textual gradients are local and sharpen a wrong strategy rather than replacing it; LATS/ToT widen the action space but never update the policy. The gap is a mechanism that escalates from tactical to strategic correction within one episode, without demonstrations.

Method

Three components operate over a single episode:

Setup: A per-step LLM evaluator E scores each transition (obs, action, next-obs, task goal) on [0,10] into a rolling window Wt of the last m=5 scores. A TODO-list decomposition tracks task state.

Routing rule (Eq. 3): Deterministic on Wt and a cooldown counter ct. - SLOW: if ct=0 and all m scores in Wt < θlow=4 (five consecutive low-progress steps) - COOL: if ct > 0 (decrement counter; no updates) - FAST: otherwise

Fast process (∼85% of steps): Every k=3 steps, computes a textual gradient gt over the most recent k trajectory tuples (TextGrad-style) and applies it to the natural-language policy πt. Local, tactical: catches recoverable action errors without causal reasoning.

Slow process (∼15% of steps): On activation, an LLM performs Reflexion-style causal diagnosis over the window, producing a plan ρt (1–3 high-level subgoals naming the root cause and corrective sequence). Sets cooldown ct←c=5 to protect plan execution from gradient interference.

Policy merge (Eq. 6): Priority plan ≻ gradient ≻ base policy. No averaging of contradictory instructions; lower-priority signals are discarded. Policy is bounded in size (empirically 150–520 tokens; capped working memory of 10 tuples).

Each slow activation emits three observable artifacts: reproducible trigger (consecutive-low-score sequence), causal diagnostic dt, and verified fix ρt.

Key Contributions

  • Progress-gated dual-process routing: deterministic rule distinguishing tactical-fixable failures (FAST) from strategy-level stalls (SLOW) within a single episode
  • Demo-free in-episode learning: 88.1%±2.0 (GPT-5) and 75.4%±2.2 (Qwen-3-8B) on ALFWorld 134 tasks, n=10 seeds, no demonstrations
  • Super-additive synergy: combined architecture gains +12.0pp (GPT-5) and +6.8pp (Qwen-3-8B) beyond the sum of individual component gains
  • Hyperparameter robustness: 3-parameter sensitivity sweep (k, m, θlow) yields worst-case 84.3%, a 3.8pp range across all tested settings
  • Failure decomposition with falsifiable scope: failure modes localized to world-knowledge (21/33), receptacle-navigation (8/33), evaluator hallucination (4/33); 5.2pp gap to 1-shot ReflAct attributed to two specific categories (Heat, Examine)
  • Full release of code, prompts, per-seed logs, and sensitivity sweeps

Results

  • ALFWorld, GPT-5, n=10 seeds, no demos: 88.1%±2.0 vs. zero-shot 46.3% (+41.8pp, p<10⁻⁶)
  • ALFWorld, Qwen-3-8B, n=10 seeds, no demos: 75.4%±2.2 vs. zero-shot 35.1% (+40.3pp, p<10⁻⁶)
  • Cross-model gain difference 1.5pp, p≈0.13 (within seed noise; architecture gain is model-independent)
  • vs. compute-matched 1-shot baselines on Qwen-3-8B:
  • beats LATS (72.7%): +2.7pp at ∼30% lower compute (p≈0.01)
  • beats ToT (69.7%): +5.7pp at equivalent compute (p<10⁻⁴)
  • beats Self-Refine (68.7%): +6.7pp (p<10⁻⁵)
  • Does not match 1-shot ReflAct (80.6%): 5.2pp gap localized to Heat and Examine categories
  • Step-budget scaling (GPT-5): 56.0% at 5 steps → 76.1% at 10 → 88.1% at 15 → 90.3% at 20 (diminishing returns beyond 15)
  • TextWorld (9 tasks, GPT-5, preliminary): 56%→89% (+33pp)
  • OSWorld (20 tasks, preliminary): 14/20→16/20
  • Cost efficiency: ReflexGrad achieves lowest calls/pp gained (1.4 calls/pp) vs. Reflexion-only (3.7) and TextGrad-only (1.7)

Limitations

  • Demo-knowledge gap: 5.2pp below 1-shot ReflAct because demonstrations transfer tacit verb-receptacle world knowledge (heat→microwave, examine→lamp) that environmental feedback cannot surface within the step budget
  • Cross-domain evaluation is preliminary: TextWorld (n=9) and OSWorld (n=20) are applicability probes, not full statistical benchmarks; WebShop and Mind2Web are identified as natural next targets
  • GPT-5 reproducibility: GPT-5 runs are not exactly reproducible; the falsifiable contribution is the Qwen-3-8B checkpoint
  • Evaluator hallucination: ∼3% false-positive rate in per-step scoring; sustained false-positive sequences account for 4/33 failures on middle seed
  • Single-episode only: no cross-episode learning; ExpeL-style knowledge accumulation across tasks is out of scope
  • Author-implemented baselines: LATS, ToT, Self-Refine are author re-implementations on ALFWorld; sanity-checked on original benchmarks but not peer-validated on this domain

Relevance to Agentic AI / LLM Agents

ReflexGrad directly attacks within-episode recovery—a key open problem for agents deployed in interactive, step-limited environments where multi-trial loops are impractical. The progress-gated dual-process router provides a principled, training-free mechanism for distinguishing when local gradient-style self-correction is sufficient versus when the agent needs causal replanning, filling the gap between Reflexion's cross-trial loop and TextGrad's local refinement. The demo-free regime is significant: it shows that ∼88% ALFWorld performance is achievable through architectural routing rather than few-shot knowledge injection, which has direct implications for agent deployment without human-curated demonstrations. The super-additive synergy finding (+12pp beyond component sums) suggests that combining temporal scales of self-correction is a productive axis for future agent architectures.