Skip to content

AgentGym-RL: An Open-Source Framework to Train LLM Agents for Long-Horizon Decision Making via Multi-Turn RL

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Fudan/Shanghai AI Lab; unified open-source multi-turn RL framework for LLM agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AgentGym-RL is an open-source, modular RL framework for training LLM agents on long-horizon multi-turn decision-making tasks across diverse real-world environments. It introduces ScalingInter-RL, a staged curriculum that progressively expands the interaction horizon during RL training to avoid collapse. A 7B model trained with this framework matches or surpasses OpenAI o3 and Gemini-2.5-Pro across 27 tasks.

Problem

The open-source community lacks a unified RL framework for training LLM agents from scratch across diverse, realistic interactive environments. Existing work is limited by narrow task complexity and insufficient environmental diversity. Moreover, directly applying RL with long interaction horizons causes training instability and collapse due to redundant, repetitive actions.

Method

AgentGym-RL provides a decoupled three-module architecture (environment, agent, training) where each environment is an independent HTTP server supporting parallel replicas. The training module unifies online RL algorithms (PPO, GRPO, RLOO, REINFORCE++) and offline methods (SFT, DPO) under a single interface. The core training innovation, ScalingInter-RL, formulates training as maximizing expected terminal reward \(J(\theta) = \mathbb{E}_{\tau \sim \pi_\theta}[r(\tau)]\) subject to a monotonically growing horizon schedule \(\{h_1 < h_2 < \cdots < h_n\}\), updated every \(\Delta\) steps by \(h_{t+1} = h_t + \delta_h\). Training begins with short horizons (exploitation of simple skills), then progressively expands, allowing the agent to discover deeper exploration strategies without collapsing.

Key Contributions

  • Open-source unified RL framework covering 5 scenario categories (web navigation, deep search, digital games, embodied control, scientific tasks) spanning 27 tasks
  • ScalingInter-RL: progressive horizon curriculum that stabilizes long-horizon RL training
  • Empirical demonstration that post-training and test-time compute scaling outperforms naive model-size scaling for agentic tasks
  • Engineering infrastructure: subprocess-based parallelism, memory-leak fixes, trajectory visualization UI, turnkey replication scripts

Results

  • ScalingInter-7B achieves 61.8% average success rate across 5 scenarios, surpassing Llama3.1-70B (46.9%) and Qwen2.5-72B (42.8%) — a 7B model outperforming 10× larger models
  • Qwen-2.5-7B with RL training gains +33.65 points on average over its base, matching or exceeding OpenAI o3 and Gemini-2.5-Pro
  • ScalingInter-7B Deep Search overall: 38.3% vs. OpenAI o3 at 49.5% (best proprietary) and Qwen3-8B base at 25.3%
  • SciWorld: score jumps from 1.50% to 50.50% (+49 points) with RL; WebArena improved by >15 points with ScalingInter-RL
  • TextCraft: ScalingInter-RL boosts score by ~50 points, achieving state-of-the-art
  • ScalingInter-RL Pass@2 on SciWorld surpasses all baselines' Pass@64
  • GRPO consistently outperforms REINFORCE++ (e.g., 3B-GRPO: 25.75% vs. 7B-REINFORCE++: 24.00% on Deep Search)
  • ScalingInter-RL is robust to hyperparameters (interaction turn lists, stage transition frequency), with performance ranging 36.8–39.1% under ablation

Limitations

  • Gains from RL are more limited in open-ended environments (WebArena, Deep Search) compared to rule-based environments (BabyAI, TextCraft, SciWorld) due to task complexity and noisy rewards
  • Backbone models limited to Qwen-2.5-3B/7B; larger backbone experiments not the primary focus
  • GRPO's advantage over REINFORCE++ is explained but no theoretical grounding is provided for the horizon increment schedule \(\delta_h\)
  • Long-horizon RL still faces instability without the curriculum; the curriculum's sensitivity to initial horizon and increment choices is not fully characterized beyond the ablation table

Relevance to Agentic AI / LLM Agents

This work directly addresses one of the central open problems in LLM agent training: how to apply RL stably across diverse, long-horizon interactive environments without task-specific tuning. The ScalingInter-RL curriculum is a practically important finding — it shows that naively scaling interaction turns causes collapse, but a monotonic horizon schedule resolves this. The finding that post-training compute and test-time interaction scaling outperform parameter scaling is a concrete, benchmark-grounded datapoint that reframes architectural priorities for the field. The fully open-sourced framework, covering environments from web navigation to embodied control, lowers the barrier for the community to run controlled comparisons of RL algorithms on agentic tasks.