Skip to content

Meta-RL Induces Exploration in Language Agents

🕒 Published (v1): 2025-12-18 18:22 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LAMER applies meta-reinforcement learning (Meta-RL) to LLM agent training, replacing single-episode RL with a cross-episode objective that explicitly incentivizes exploration in early episodes and exploitation in later ones. In-context policy adaptation via self-reflection serves as the inner loop, requiring no gradient updates at test time. LAMER achieves 11–19% absolute gains over the strongest RL baselines (GiGPO) on Sokoban, MineSweeper, and Webshop.

Problem

Standard RL-trained LLM agents optimize single-episode returns, causing them to converge to deterministic, low-diversity policies that fail to actively explore unfamiliar environments or efficiently adapt from trial-and-error feedback across multiple attempts. Prior work on inducing exploration either targets single-turn non-agentic tasks or relies on offline imitation rather than online active exploration.

Method

LAMER structures training as multi-episode trials (N=3 episodes by default). Two components:

  1. Cross-episode training objective: Each trial T = (τ⁰, τ¹, …, τᴺ⁻¹) is rolled out sequentially. The return for action at episode n is a discounted sum of within-episode return plus future-episode returns weighted by a trajectory discount factor γ_traj (default 0.6). This cross-episode credit assignment incentivizes informative early-episode behavior even when immediate reward is zero.

  2. In-context policy adaptation via self-reflection: After each episode, the agent generates a textual reflection summarizing what happened and planning the next attempt. This reflection (and optionally trajectory history) forms inter-episode memory H^(n), conditioning the next episode's policy without gradient updates. Reflection generation is itself trained via the reward signal from the subsequent episode.

The objective is optimized with standard policy gradient (GiGPO by default; compatible with PPO/GRPO/RLOO). At test time, the agent uses the same reflection-based in-context adaptation—no additional finetuning required.

Key Contributions

  • First application of a Meta-RL framework to LLM agent training (claimed).
  • Cross-episode discounted return formulation with γ_traj controlling exploration–exploitation balance.
  • Trained self-reflection as the inner-loop adaptation mechanism; ablation shows reflection-only memory (pass@3: 56.4% Sokoban, 80.5% MineSweeper, 92.8% Webshop) outperforms both trajectory-only and trajectory+reflection variants.
  • Empirical demonstration that Meta-RL preserves trajectory diversity (higher entropy) relative to RL while simultaneously improving success rates.
  • Generalization results: LAMER trained on easier tasks outperforms RL on harder difficulty levels (+10% Sokoban, +5% MineSweeper) and on OOD task categories (+23% on ALFWorld Cool, +14% on Pick2).

Results

  • Sokoban (pass@3): LAMER 55.9% vs. GiGPO 44.1% (+11.8pp); Zero-shot 12.9%.
  • MineSweeper (pass@3): LAMER 74.4% vs. GiGPO 55.1% (+19.3pp).
  • Webshop (pass@3): LAMER 89.1% vs. GiGPO 75.2% (+13.9pp).
  • Test-time scaling: LAMER pass@1→pass@3 gain on Sokoban is 13.5pp, vs. <5pp for all RL/prompting baselines.
  • ALFWorld OOD (Cool): LAMER 81.0% vs. RL 58.1% (+22.9pp); (Pick2): 50.2% vs. 36.0% (+14.2pp).
  • Training cost: ~2× wall-clock time vs. RL due to sequential episode sampling; same trajectory budget per gradient step.
  • Base model: Qwen3-4B (non-thinking mode); validated also on Llama3.1-8B-Instruct.

Limitations

  • Sequential episode rollouts within a trial reduce parallelism and roughly double training time compared to standard RL.
  • γ_traj requires per-environment tuning; optimal value varies (0.6 for Sokoban/Webshop, 0.9 for MineSweeper).
  • Generalization is demonstrated within the same task family (harder Sokoban/MineSweeper) or closely related household tasks (ALFWorld); transfer to completely novel environment types is not shown.
  • Experiments use a 4B-parameter base model; scaling behavior with larger models is not studied.
  • Memory buffer management (truncating H^(n)) trades off context efficiency against adaptation quality; optimal strategy not fully characterized.

Relevance to Agentic AI / LLM Agents

LAMER directly addresses a core bottleneck for autonomous agents: the inability of RL-trained LLMs to systematically explore when deployed in novel or harder environments. By framing multi-attempt task-solving as Meta-RL with in-context adaptation, it offers a training recipe that is architecturally compatible with any policy-gradient optimizer and requires no test-time finetuning—making it practically deployable. The self-reflection inner loop connects to the Reflexion line of work but grounds reflection training in actual downstream reward rather than heuristic prompting, providing a principled path toward agents that self-improve across episodes. The stronger OOD generalization results are especially relevant to researchers building agents that must handle distribution shift in real-world deployment settings.