Skip to content

Generalizable End-to-End Tool-Use RL with Synthetic CodeGym

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

End-to-end RL for generalizable tool use; moves beyond SFT for agentic training

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CodeGym is a scalable framework that converts static coding problems into verifiable multi-turn tool-use RL environments by extracting atomic functions as callable tools. Models trained via GRPO on CodeGym's 80k+ task configurations achieve consistent out-of-distribution gains across tool-use, multi-turn interaction, and reasoning benchmarks. Qwen2.5-32B-Instruct gains +8.7 points on \(\tau\)-Bench and QwQ-32B gains +4.6 points on \(\tau^2\)-Bench.

Problem

SFT over static trajectories and task-specific RL environments both generalize poorly: hand-crafted demonstrations explore limited workflows, and narrow RL setups (e.g., code-debugging-only, search-only) overfit to their domain. A scalable, general-purpose RL environment for training transferable tool-use behaviors in LLM agents is missing.

Method

CodeGym synthesizes POMDPs \(\mathcal{E} = \langle S, A, T, R, O \rangle\) from competitive programming problems (sourced from KodCode). An LLM (Seed-1.6-Thinking) extracts atomic, reusable functions from coding solutions into callable tools; the agent must invoke these tools sequentially to solve the task rather than writing full code. A two-stage pipeline handles correctness:

  1. Gym Synthesis: prompts the LLM to produce tool docstrings, partial-state Observe(), and a Done(answer) terminal action in OpenAI Gym format.
  2. Gym Verification: generates unit-test inputs/outputs from the ground-truth solution, then uses pass@10 on LLM-generated oracle functions to confirm solvability.

Quality filters remove tasks with fewer than \(T_{\min}=10\) or more than \(T_{\max}=256\) tool calls, fewer than 4 distinct tools, and tasks where Qwen2.5-32B passes >25% of runs (too easy). A trial-then-overwrite mechanism serializes environment state before each tool call, executes in a subprocess, and rolls back on failure—preventing server crashes. Long-CoT models receive a difficulty-augmented variant that withholds information to prevent pure-reasoning shortcuts. GRPO with batch size \(512 \times 8\) trains short-CoT (Qwen2.5 7B/14B/32B/72B) and long-CoT (QwQ-32B) models.

Key Contributions

  • CodeGym framework: converts static coding problems into interactive, verifiable, multi-turn POMDP tool-use environments at scale (13k environments, 80k+ task configurations, avg. 6.52 tools and 44.07 steps per task).
  • End-to-end RL pipeline: distributed CPU-bound environment server decoupled from GPU rollouts; trial-then-overwrite for robustness.
  • OOD generalization: demonstrates that RL on code-derived environments transfers to semantically distinct benchmarks (retail/airline tool-use, household navigation, logical reasoning).
  • Difficulty augmentation for long-CoT models to prevent reasoning-without-tool-use shortcuts.

Results

All gains are absolute accuracy improvements over the untuned base model.

  • \(\tau\)-Bench (Ï„-airline / Ï„-retail):
  • Qwen2.5-7B: +4.5 / +3.1
  • Qwen2.5-14B: +3.7 / +7.2
  • Qwen2.5-32B: +4.4 / +13.0 (retail best among short-CoT)
  • Qwen2.5-72B: +6.0 / +7.8
  • QwQ-32B: +5.6 / +5.3
  • \(\tau^2\)-bench: Qwen2.5-32B +6.0, QwQ-32B +4.6
  • ALFWorld (multi-turn, household navigation): Qwen2.5-32B +14.0, Qwen2.5-14B +13.6
  • ZebraLogic (reasoning): Qwen2.5-32B +4.8, QwQ-32B −3.3 (slight regression)
  • MMLU-Pro (reasoning): gains of 0.9–1.2 for short-CoT; 0.0 for QwQ-32B
  • Average across all benchmarks: Qwen2.5-32B +7.3, QwQ-32B +2.4
  • Training and in-domain validation curves track closely, indicating minimal overfitting.

Limitations

  • ZebraLogic performance degrades for QwQ-32B (−3.3), suggesting long-CoT reasoning may be partially disrupted by tool-use RL training.
  • \(\tau^2\)-bench results for Qwen2.5-14B show a marginal regression (−1.0), indicating inconsistency at smaller scales.
  • Single seed reported for 72B and QwQ-32B models due to compute constraints—stability across seeds is unverified at that scale.
  • CodeGym is derived from competitive programming (KodCode); it is unclear how coverage varies across tool-use domains underrepresented in competitive programming (e.g., API chaining, GUI interaction).
  • Difficulty augmentation is applied heuristically; its interaction with data scaling is not ablated.
  • The paper does not report wall-clock training time or token cost, making practical reproducibility harder to assess.

Relevance to Agentic AI / LLM Agents

CodeGym directly attacks the core generalization problem in LLM agent training: how to build RL environments diverse enough that learned tool-use policies transfer to unseen workflows. The key insight—that code execution logic structurally mirrors real-world agentic workflows—offers a principled, scalable bridge between synthetic training and real deployment. The consistent OOD gains across \(\tau\)-Bench (customer-service tool-use), ALFWorld (embodied navigation), and reasoning benchmarks validate that the approach learns transferable interaction strategies rather than domain-specific shortcuts. For researchers building agents that must orchestrate tools dynamically, CodeGym provides both a reusable training infrastructure and empirical evidence that RLVR on structured synthetic environments is a viable path to generalizable agentic behavior.