GEM: A Gym for Generalist LLMs

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Training environment for generalist LLMs via experience-based learning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GEM (General Experience Maker) is an open-source environment simulator analogous to OpenAI-Gym but designed for RL training of LLM agents on diverse multi-turn, long-horizon tasks. It provides a standardized agent-environment interface, asynchronous vectorized execution, modular tool support, and integration with five popular LLM RL training frameworks. Alongside the framework, the authors introduce REINFORCE with Return Batch Normalization (ReBN), a simple yet broadly effective multi-turn RL algorithm that outperforms GRPO and matches PPO without a critic.

Problem

Current RL research for LLMs overwhelmingly targets single-turn tasks (math QA, data retrieval), and algorithms optimized for that setting (notably GRPO) are fundamentally incompatible with full multi-turn RL: they either degenerate to contextual bandits (action = whole interaction), fix \(\gamma = 1\), or face combinatorial explosion in per-turn advantage estimation. No standardized, unified environment suite existed for benchmarking and training agentic LLMs on complex, multi-turn, long-horizon tasks.

Method

GEM adopts the OpenAI-Gym reset()/step() API and provides: - Environments across seven task categories: Math, Math-with-image, Code, Game (language games up to 100+ turns), QA, ReasoningGym (100+ single-turn verifiable tasks via unified interface), and Terminal (containerized shell). - Tools as modular wrappers: Python interpreter, web Search, and MCP-compatible external servers; these convert single-turn tasks into multi-turn tool-use environments. - Asynchronous vectorized execution with autoreset for high-throughput episode collection. - Observation/action wrappers controlling how history is exposed to the agent.

The baseline RL algorithm is REINFORCE with Return Batch Normalization (ReBN):

\[J_{\text{REINFORCE+ReBN}}(\theta) = \frac{1}{N} \sum_{n=1}^{N} \sum_{t=0}^{T^{(n)}-1} A_{\text{ReBN},t}^{(n)} \log \pi_\theta(a_t^{(n)} | s_t^{(n)})\]

where \(A_{\text{ReBN},t}^{(n)} = (G_t^{(n)} - \text{mean}(\mathcal{G})) / \text{std}(\mathcal{G})\) and \(\mathcal{G}\) is the set of all per-transition returns in the batch. This normalizes returns across the entire batch of transitions (not per-initial-state group as in GRPO), enabling dense per-turn rewards and arbitrary \(\gamma \leq 1\).

Key Contributions

• Open-source GEM framework: standardized multi-turn LLM RL environment suite with 24+ environments spanning games, reasoning, code, math, QA, and terminal tasks.
• Asynchronous vectorized execution with autoreset for efficient batched episode collection.
• Modular tool integration (Python, Search, MCP) as environment wrappers.
• Single-file training scripts integrating GEM with five LLM RL frameworks: Oat, Verl, OpenRLHF, ROLL, and RL2.
• REINFORCE+ReBN algorithm: compatible with dense per-turn rewards and \(\gamma < 1\), unlike GRPO; no critic needed unlike PPO.
• Apples-to-apples benchmarking of PPO, GRPO, REINFORCE, and REINFORCE+ReBN across 8 environments.
• Empirical analyses of discount factor effects, tool integration benefits, and cross-environment generalization.

Results

• Algorithm benchmarking (8 environments, Qwen3-Base): REINFORCE+ReBN outperforms or matches PPO and GRPO in all evaluated environments; GRPO fails on multi-turn tasks with dense rewards (GuessTheNumber, Sudoku) due to constant advantage across turns.
• Discount factor: With \(\gamma = 0.9\) on GuessTheNumber, the agent converges to the optimal binary search strategy (~\(\log_2(50) \approx 5.6\) turns); larger \(\gamma\) leads to suboptimal longer episodes.
• Math tool integration (Qwen3-4B-Base, 5 benchmarks):
• Base (no tool): 35.3% avg
• Base + RL (no tool): 41.4% avg
• Base + RL (with Python tool): 49.8% avg
• AIME24 specifically improves from 10.0 (base) to 30.0 (RL + tool).
• QA tool integration (Qwen3-4B, 7 datasets):
• Base (no tool): 10.2% avg
• Base + RL (no tool, mixed env): 23.9% avg
• Base + RL (with Search tool, mixed env): 45.5% avg
• Cross-environment generalization: Training on game:sudoku-v0-easy yields generalization gains on rg:mini_sudoku, rg:needle_haystack, and rg:circuit_logic.

Limitations

• ReBN is still a simple variance-reduction technique; it does not learn a value function and may have suboptimal convergence in highly sparse or complex reward structures (noted for vanilla REINFORCE on Sudoku; ReBN mitigates but does not fully solve this).
• PPO with a well-trained critic outperforms ReBN on the hardest long-horizon environments (Sudoku), but PPO's critic is difficult to learn robustly (poor performance on Minesweeper).
• Math benchmark scores are sensitive to grading code; the authors caution against cross-codebase absolute comparisons.
• Terminal and multi-agent environments are present but receive limited empirical analysis in the main paper (relegated to appendices).
• No cross-framework apples-to-apples training performance comparison is provided; only integration feasibility is demonstrated.

Relevance to Agentic AI / LLM Agents

GEM directly addresses a key bottleneck in agentic LLM research: the absence of a unified, extensible environment suite for training and evaluating agents on multi-turn, tool-using, long-horizon tasks. The framework makes it feasible to study the full RL problem for LLMs—with per-turn dense rewards, arbitrary discount factors, and heterogeneous tools—rather than the degenerate single-turn/contextual-bandit approximations dominant in current work. The ReBN algorithm provides a practical, drop-in baseline that handles multi-turn credit assignment without critic complexity, directly relevant to anyone training LLM agents on interactive tasks. The framework's MCP compatibility and cross-environment generalization experiments point toward important open questions in agentic transfer and compositionality.