Meta-RL Induces Exploration in Language Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Stanford (Teney, Brbic, Moor); Meta-RL induces principled exploration in LLM agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LAMER applies Meta-Reinforcement Learning to train LLM agents to actively explore environments across multiple episodes, using cross-episode credit assignment and in-context self-reflection as the inner-loop adaptation mechanism. Unlike standard single-episode RL, which converges to deterministic policies, LAMER trains agents to balance exploration and exploitation by conditioning each episode on reflections from prior attempts. It achieves 11–19% absolute gains over RL baselines on Sokoban, MineSweeper, and Webshop.

Problem

Standard RL-trained LLM agents learn a fixed policy and fail to actively explore at test time: they converge toward deterministic, low-diversity behaviors and cannot efficiently adapt from trial-and-error feedback in novel or harder environments. Prior work on inducing exploration either targets single-turn non-agentic settings or relies on offline imitation, not active online exploration.

Method

LAMER structures training as trials of \(N\) sequential episodes on the same task instance. The cross-episode return for action \(a_t^{(n)}\) at step \(t\) in episode \(n\) is:

\[G_t^{(n)} = g_t^{(n)} + \sum_{m=n+1}^{N-1} \gamma_{\text{traj}}^{m-n} g_0^{(m)}\]

where \(g_t^{(n)}\) is the within-episode discounted return and \(\gamma_{\text{traj}} \in [0,1]\) is a cross-episode discount that controls the exploration–exploitation trade-off. The Meta-RL objective maximizes \(J(\theta) = \mathbb{E}_{\mathcal{T} \sim \pi_\theta}[G_0^{(0)}]\), optimized via standard policy gradient (GRPO/GiGPO/PPO).

The inner-loop adaptation is implemented as in-context self-reflection: after each episode, the agent is prompted to generate a textual reflection summarizing failures and proposing a plan, which is prepended to the next episode's context. The reflection step itself is trained via the reward of the subsequent episode. At test time, no gradient update occurs; adaptation is purely in-context.

Key Contributions

• First application of a Meta-RL framework to LLM agent training, providing a principled mechanism for inducing exploration.
• Cross-episode training objective with \(\gamma_{\text{traj}}\) for multi-episode credit assignment, compatible with standard RL optimizers (PPO, GRPO, GiGPO).
• In-context policy adaptation via trained self-reflection, serving as the Meta-RL inner loop without gradient updates at test time.
• Empirical demonstration that Meta-RL preserves trajectory diversity (higher entropy) relative to RL, quantifying the exploration-exploitation balance.
• Generalization to harder (more boxes/mines) and out-of-distribution (ALFWorld Cool, Pick2) tasks beyond the training distribution.

Results

• Sokoban (pass@3): LAMER 55.9% vs. GiGPO (best RL) 44.1% — +11.8 pp; pass@1→pass@3 gain of 13.5% for LAMER vs. <5% for all baselines.
• MineSweeper (pass@3): LAMER 74.4% vs. GiGPO 55.1% — +19.3 pp.
• Webshop (pass@3): LAMER 89.1% vs. GiGPO 75.2% — +13.9 pp.
• ALFWorld OOD (Cool): LAMER achieves ~23 pp gain over RL; OOD (Pick2): ~14 pp gain over RL.
• Generalization to harder tasks: On hardest Sokoban/MineSweeper settings, LAMER retains a ~10 pp and ~5 pp gap over RL, respectively.
• Trajectory diversity: Meta-RL maintains higher entropy than RL across Sokoban, MineSweeper, and Webshop without sacrificing performance.
• Base model: Qwen3-4B; also validated on Llama3.1-8B-Instruct (Appendix D.1).

Limitations

• Evaluated only in text-based environments; multimodal extension is stated as future work.
• \(N=3\) episodes per trial; the sensitivity of performance to larger \(N\) is not fully explored in the main results.
• Training compute is matched to RL (same total trajectory count), but multi-episode sequential rollout increases wall-clock training time.
• Self-reflection quality depends on the base model's instruction-following capability; weaker models may produce uninformative reflections.
• Generalization experiments are limited to two OOD categories in ALFWorld; broader distribution shift scenarios are not tested.

Relevance to Agentic AI / LLM Agents

LAMER directly addresses one of the core open problems in LLM agents: the inability of standard RL-trained agents to explore and adapt at test time, which is critical for deployment in novel environments. By framing multi-episode interaction as Meta-RL and using in-context self-reflection as the gradient-free inner loop, it provides a scalable training recipe applicable to any environment with episodic feedback. The cross-episode credit assignment signal (\(\gamma_{\text{traj}}\)) is a concrete mechanism for aligning agent behavior with long-horizon, trial-and-error objectives, which is broadly applicable to tool-use, web navigation, and embodied agents. The result that Meta-RL improves test-time scaling (pass@k) connects directly to the growing literature on allocating test-time compute for agentic tasks.