Skip to content

Scaling Agent Learning via Experience Synthesis

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Experience synthesis for efficient RL agent training without costly rollouts

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DreamGym is a unified RL framework that replaces costly real-environment rollouts with a reasoning-based experience model — an LLM that synthesizes state transitions and reward signals via chain-of-thought — enabling scalable online RL for LLM agents. It pairs this with an experience replay buffer and a curriculum task generator that selects tasks by reward entropy. On WebArena (not RL-ready), it outperforms all baselines by >30%; in sim-to-real transfer it achieves >40% gains over from-scratch real-env RL using <10% of the real data.

Problem

Online RL for LLM agents is blocked by four compounding barriers: (1) real-environment rollouts are prohibitively expensive and slow; (2) most environments offer only static, limited task sets inadequate for goal-conditioned RL; (3) reward signals in dynamic environments (web pages, GUIs) are sparse, noisy, and unstable; (4) RL-ready infrastructure (Docker, VMs) is heterogeneous and engineering-heavy. Together these prevent scalable experience collection and generalization.

Method

DreamGym centers on three components:

Reasoning Experience Model (\(M_\text{exp}\)). Rather than simulating raw pixels or HTML, it operates in an abstract textual state space. At each step it receives: current state-action pair, full interaction history \(\{(s_i, a_i)\}_{t=0}^T\), task instruction \(\tau\), and top-\(k\) demonstrations retrieved from the replay buffer via cosine similarity in a semantic embedding space. It produces the next state and reward via explicit chain-of-thought: $\((s_{t+1}, r_{t+1}) = M_\text{exp}\!\left(R_t \mid \{(s_i,a_i)\}_{i=0}^t,\, \{d_j\}_{j=1}^k,\, \tau\right)\)$ \(M_\text{exp}\) is trained via SFT on annotated offline trajectories with a joint objective over reasoning traces and next-state prediction: $\(\mathcal{L}_\text{SFT} = \mathbb{E}\!\left[-\log P_\theta(R_t^* \mid s_t, a_t, H_t, D_k) - \log P_\theta(s_{t+1} \mid s_t, a_t, R_t^*, H_t, D_k)\right]\)$

Experience Replay Buffer. Seeded with offline public trajectory data, continuously updated with freshly synthesized rollouts. Top-\(k\) retrieval by semantic similarity grounds future predictions and suppresses hallucination.

Curriculum Task Generator (\(M_\text{task}\), sharing parameters with \(M_\text{exp}\)). Selects seed tasks by group-based reward entropy — variance of outcome rewards across \(n\) rollouts: $\(V_\tau = \frac{1}{n}\sum_{i=1}^n (r^i - \bar{r})^2\)$ High-variance tasks are feasible yet challenging (agent sees mixed successes/failures), maximizing information gain for credit assignment. \(M_\text{task}\) generates progressively harder variations of these tasks.

Policy training uses standard GRPO or PPO over synthetic rollouts. The DreamGym-S2R variant first pretrains the policy purely on synthetic data, then transfers to real-environment RL with a lightweight state-space alignment step.

Key Contributions

  • First unified framework for synthesizing RL training experience for LLM agents without requiring RL-ready real environments.
  • Reasoning-based experience model operating in abstract textual state space, trained sample-efficiently from small public trajectory datasets.
  • Reward-entropy-driven curriculum task generation that co-evolves task difficulty with agent capability.
  • Active replay buffer that co-evolves with the agent to keep synthetic rollouts policy-aligned.
  • Sim-to-real transfer strategy (DreamGym-S2R) as a scalable warm-start for real-environment RL.
  • Theoretical lower bound on policy improvement in real environments when trained on purely synthetic DreamGym experiences (under trust-region assumptions).

Results

  • WebArena-Lite (not RL-ready): DreamGym outperforms all baselines (SFT, DPO, and traditional RL variants) by >30% across Llama-3.2-3B, Llama-3.1-8B, and Qwen-2.5-7B; e.g., DreamGym-S2R reaches 49.1/73.3/79.9 vs. traditional GRPO's 47.0/72.9/81.1 — achieved with 5K real transitions vs. 80K.
  • WebShop: DreamGym (0 real data) matches traditional GRPO/PPO (80K real transitions); DreamGym-S2R (5K) improves further, e.g., Qwen-2.5-7B reaches 72.1 (GRPO) and 73.7 (PPO) vs. traditional 66.1/68.1.
  • ALFWorld: DreamGym-S2R with 5K real samples reaches 13.9/10.9/13.9 across the three backbones under GRPO, matching or exceeding traditional GRPO at 80K (7.3/6.1/6.1) by ~2×.
  • DreamGym-S2R achieves >40% improvement over from-scratch real-env RL while using <10% of external interaction data.
  • All results use experience model trained from Llama-3.1-8B-Instruct.

Limitations

  • Experience model fidelity depends on quality and coverage of offline seed trajectories; rare or novel state transitions may be poorly generalized.
  • Abstract state-space design requires domain-specific engineering of the state representation (e.g., rule-based mapping or lightweight fine-tuning) for sim-to-real transfer alignment.
  • Experiments are limited to three benchmarks (WebShop, ALFWorld, WebArena-Lite) and relatively small backbone models (3B–8B); scalability to larger models and more diverse domains is not fully demonstrated.
  • Outcome-based sparse reward (\(r=1\) only at final success) may still impede learning on very long-horizon tasks even within the synthetic environment.
  • The curriculum task generator shares parameters with the experience model, potentially coupling training failures across both roles.

Relevance to Agentic AI / LLM Agents

DreamGym directly addresses the sample-efficiency and infrastructure bottlenecks that have made RL impractical for general-purpose LLM agents, offering a path to agent self-improvement without expensive real-environment deployments. The reasoning experience model as a world surrogate is a conceptually important shift: it sidesteps fidelity requirements by targeting "causally grounded and diverse" rather than "realistic" interactions — a useful design philosophy for the field. The sim-to-real transfer result (>40% gain, <10% real data) is particularly relevant to practitioners deploying agents in high-cost or safety-constrained environments. This work connects to the broader line of model-based RL and world-model research (Dreamer, WebDreamer) but tailors it specifically to the multi-step, language-grounded, tool-use setting that defines modern agentic systems.