Skip to content

Generalizable End-to-End Tool-Use RL with Synthetic CodeGym

🕒 Published (v1): 2025-09-22 03:03 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CodeGym is a scalable pipeline that converts static coding problems into diverse, verifiable multi-turn tool-use environments for reinforcement learning, enabling LLMs to acquire generalizable agent behaviors. By abstracting reusable code logic into callable tools and training via GRPO, models trained on CodeGym transfer to semantically distinct OOD benchmarks. Qwen2.5-32B gains +8.7 points on \(\tau\)-Bench and +14.0 points on ALFWorld.

Problem

Current LLM agent training relies on SFT over static trajectories or RL on narrow, domain-specific tasks, producing agents that are brittle under distribution shift (new tools, unseen workflows). No scalable, general-purpose, verifiable RL environment exists for cultivating transferable tool-use behaviors.

Method

CodeGym ingests competitive programming problems (sourced from KodCode) and transforms each into a POMDP environment \(E = \langle S, A, T, R, O \rangle\) via a two-stage pipeline: 1. Gym Synthesis: An LLM (Seed-1.6-Thinking) extracts atomic, reusable functions from the coding solution and rewrites them as documented callable tools; the environment is packaged in OpenAI Gym format with partial observability (some task inputs hidden) to prevent shortcutting. 2. Gym Verification: Unit test inputs spanning difficulty levels are generated; \(K{=}10\) candidate solution functions are sampled (pass@K); an environment is accepted only if at least one candidate passes all unit tests within time/memory limits.

Quality filters require \(T_{\min}{=}10\) and \(T_{\max}{=}256\) tool calls per trajectory and at least 4 distinct tools. Configurations with \(>25\%\) baseline pass rate are removed. A difficulty augmentation step modifies task initializations to force long-CoT models to actually invoke tools rather than reason through the full answer. Training uses GRPO with a distributed rollout framework: a CPU-bound environment server manages state via a trial-then-overwrite mechanism (pickle-snapshot, subprocess execution, rollback on failure), decoupled from GPU rollout workers.

Key Contributions

  • CodeGym dataset: >80k task configurations across 13k environments, averaging 6.52 tools and 44.07 steps per trajectory — largest agent training set reported against compared baselines.
  • Scalable synthesis pipeline: LLM-driven extraction + verification (pass@K unit-test oracle) converts arbitrary coding problems into stable, solvable interactive environments automatically.
  • Distributed RL infrastructure: CPU/GPU-decoupled rollout with trial-then-overwrite for robustness against erroneous LLM tool calls.
  • OOD generalization evidence: Training on code-derived environments transfers to tool-use (\(\tau\)-Bench/\(\tau^2\)-Bench), embodied navigation (ALFWorld), and reasoning (ZebraLogic, MMLU-Pro) benchmarks.

Results

  • Qwen2.5-32B-Instruct → Qwen2.5-32B-CodeGym: +8.7 on \(\tau\)-retail (26.8→31.2 for retail, 41.4→54.4 on \(\tau^2\)-bench), +14.0 on ALFWorld (66.8→80.8).
  • Qwen2.5-72B-CodeGym: average improvement +4.1 across all benchmarks.
  • QwQ-32B-CodeGym (long-CoT): +5.6 on \(\tau\)-retail, +4.6 on \(\tau^2\)-bench; slight regression on ZebraLogic (−3.3).
  • MMLU-Pro degrades marginally for some models (7B: −0.3), suggesting reasoning capacity largely preserved.
  • In-domain validation curves track training curves closely, indicating limited overfitting.

Limitations

  • Reward is binary (1/0); no partial credit or process reward for multi-step tool chains.
  • 72B and QwQ results use a single seed due to compute constraints, limiting statistical reliability.
  • Code-derived environments may not capture the full diversity of real-world workflows (e.g., ambiguous user intent, API rate limits, adversarial inputs).
  • The difficulty augmentation strategy is heuristic; it may not generalize to all long-CoT model families.
  • QwQ shows regression on ZebraLogic (−3.3), suggesting potential trade-offs in pure deductive reasoning.
  • Synthesis pipeline itself uses a powerful LLM (Seed-1.6-Thinking), making environment generation costs non-trivial.

Relevance to Harnesses / Meta-Harnesses

CodeGym is a training harness that auto-generates verified, executable sub-environments from raw problem corpora — directly relevant to meta-harness design patterns where a higher-level orchestrator synthesizes and validates task instances at scale. The two-stage synthesis+verification loop with pass@K oracle checking is a concrete example of a meta-harness quality gate, analogous to how digest/trend harnesses might validate outputs before committing them. The distributed rollout controller with centralized environment server and decoupled CPU/GPU execution mirrors the architecture of multi-agent meta-harnesses that manage heterogeneous compute pools. Practitioners building training or evaluation harnesses over LLM agents can directly adapt CodeGym's POMDP templating, difficulty filtering, and trial-then-overwrite robustness mechanism.