Retrospex: Language Agent Meets Offline Reinforcement Learning Critic¶

🕒 Published (v1): 2025-05-17 03:28 UTC · Source: Arxiv · Venue: EMNLP 2024 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Retrospex is an LLM-agent framework that sidesteps context-length constraints on experience reuse by training a lightweight offline RL critic (IQL-based GRU) on collected trajectories, then combining the critic's action values with the LLM's token probabilities at inference via a dynamic rescoring weight. This decouples experience utilization from context injection, avoiding catastrophic forgetting of the base LLM while still learning from failures.

Problem¶

Prior LLM-agent approaches that leverage past experience (Reflexion, Rememberer, ExpeL) inject experiences directly into the LLM context, which is bounded by context-length limits and scales poorly. Fine-tuning the LLM on experiences risks catastrophic forgetting and is computationally expensive. Online RL is unstable in large text action spaces. The gap is: how to exploit imperfect, cross-task experience memory without modifying the LLM or expanding its context.

Method¶

Retrospex has two training stages followed by a dynamic-rescoring inference stage:

Warm-up stage: Imitation learning (IL) fine-tunes the LLM on golden expert trajectories via NLL minimization. The trained LLM then collects experience trajectories (both successes and failures) into an offline memory buffer.
Retrospection stage: An RL Critic (GRU network with ~2.7M parameters) is trained on the offline buffer using Implicit Q-Learning (IQL). IQL avoids overestimation of unseen actions by factoring out a separate state-value function $V_\phi(s)$ fit via an upper expectile loss: $$\mathcal{L}_V(\phi) = \mathbb{E}_{(s,a)\sim\mathcal{D}}[\mathcal{L}^\tau_2(Q_{\hat\theta}(s,a) - V_\phi(s))]$$ The Q-network then updates with: $$\mathcal{L}_Q(\theta) = \mathbb{E}[(r(s,a) + \gamma V_\phi(s') - Q_\theta(s,a))^2]$$
Dynamic action rescoring at inference: The LLM generates $K$ candidates via nucleus sampling, mapped to valid actions via Sentence Transformer cosine similarity. LLM log-probabilities $p$ and critic values $q$ are normalized and combined: $$\alpha(t) = \max(b,\, d^t), \quad S(a) = \alpha(t)\,p + (1-\alpha(t))\,q$$ where $d < 1$ is a per-step discount and $b$ is a lower bound, so the critic's influence grows with trajectory length while the LLM retains a floor contribution.

Key Contributions¶

Offline RL critic (IQL over GRU) trained on LLM-collected experience, decoupled from the LLM's context and weights.
Dynamic action rescoring that interpolates LLM likelihood and critic Q-values with a step-dependent weight $\alpha(t) = \max(b, d^t)$, giving the LLM primacy early and the critic primacy late.
Demonstration that a 2.7M-parameter GRU critic can improve a 770M–8B LLM agent without any LLM weight updates at inference time.
Evaluation across three text-based environments (ScienceWorld, ALFWorld, Webshop) showing consistent improvement over the IL-only base agent.

Results¶

ScienceWorld (Flan-T5-large): Retrospex achieves AS=55.98 / SR=45, vs. IL-T5 AS=48.80 / SR=36 — +7 AS, +9 SR. Outperforms GPT-4 ReAct (AS=36.43) and GPT-4 Reflexion (AS=45.34).
ALFWorld (LLaMA3-8B): SR=87.0% vs. IL-LLaMA3 SR=83.5% (+3.5 pp). Below A$^3$T round=1 (94.0%) which uses substantially more and higher-quality training data.
Webshop (LLaMA3-8B, Rememberer test set): SR=46.0 vs. IL-LLaMA3 SR=42.4 (+3.6 pp), AS=74.6 vs. 76.2 (slight regression); beats Rememberer (GPT-3.5) SR=38.0.
Webshop (AgentBoard test set): AS=77.2, SR=49.0 vs. A$^3$T round=1 AS=73.5, SR=49.0 — Retrospex matches or exceeds A$^3$T on SR with a single shared LLM.
ScienceWorld ablation by task complexity shows the retrospection gain is concentrated on medium/long-horizon tasks (SR lift: short +4.3, medium +24.3, long +12.0 reward points).

Limitations¶

The RL Critic is environment-specific; a new GRU must be trained per environment, adding a per-domain data collection and training step.
Action mapping via Sentence Transformer cosine similarity introduces a heuristic component that may fail in environments with ambiguous action semantics.
ALFWorld results fall short of A$^3$T (round=1) when that baseline has access to higher-quality and larger training sets; data quantity and quality differences complicate direct comparison.
The warm-up IL stage still requires golden expert demonstrations; the framework is not fully demonstration-free.
IQL hyperparameters ($\tau$, $\gamma$) and rescoring hyperparameters ($d$, $b$) are tuned per environment; generalization cost is not analyzed.
No evaluation on open-ended or non-text-based environments; scope is limited to discrete text-action spaces.

Relevance to Harnesses / Meta-Harnesses¶

Retrospex is an agent meta-harness pattern: it wraps an arbitrary LLM-based actor with an offline RL critic trained on the actor's own execution traces, functioning as a persistent improvement loop across runs without modifying the actor itself. This is architecturally analogous to a meta-harness that collects agent execution history into a replay buffer and trains a lightweight scoring module that reshapes the agent's decisions in subsequent runs — a form of outer-loop optimization over the agent's inner-loop policy. The dynamic rescoring mechanism ($\alpha(t)$) is an explicit meta-level controller that modulates how much the "harness layer" overrides the base agent depending on trajectory depth, directly paralleling harness designs that increase intervention strength as task difficulty or step count grows. For researchers building harnesses that learn from agent experience without fine-tuning the underlying model, Retrospex's IQL-over-GRU pattern offers a practical, low-overhead instantiation of this idea.