Skip to content

ToolVerse: Unlocking Massive Environments and Long-Horizon Tasks for Agentic Reinforcement Learning

🕒 Published (v1): 2026-07-17 06:12 UTC · Source: Arxiv · link

Why this paper was selected

ToolVerse: massive diverse environments + long-horizon tasks close the evaluation gap for real-world agentic RL

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ToolVerse is an agentic RL framework that addresses three simultaneous bottlenecks: sparse tool environments, lack of long-horizon task curricula, and coarse credit assignment. It automatically converts 422 real-world MCP toolsets (∼4438 tools) into executable training environments, synthesizes coherent multi-step tasks via a tool dependency graph, and introduces Turn-Aware Relative Advantage Estimation (TARA) for per-turn credit assignment. Training on the resulting GUST dataset yields consistent double-digit gains on three multi-turn agentic benchmarks.

Problem

Existing agentic RL systems are bottlenecked on three axes: (1) environments expose only one or two tool types (code interpreter, search engine), preventing generalization to diverse APIs; (2) there is no principled method to generate tasks that require chained, causally-dependent tool invocations across many turns; and (3) standard GRPO assigns a single scalar advantage to an entire trajectory, making it impossible to distinguish correct intermediate steps from fatal later-stage errors under sparse terminal rewards.

Method

Step 1 — Executable environment synthesis. Raw JSON MCP tool schemas \(S_i\) are refactored, paired with a mock database \(D_i\), and compiled into executable Python functions \(F_i\). Only toolsets that pass automatically generated unit tests are retained, yielding 422 environments with 5–20 tools each.

Step 2 — Graph-based long-horizon task synthesis (GUST). A Tool Dependency Graph \(G=(V,E)\) is inferred per environment via LLM: edge \(T_A \to T_B\) if \(T_B\) requires \(T_A\)'s output or must follow it logically. The Dynamic Unlocking Sampling (DUS) algorithm maintains a ready queue \(Q\) of zero-in-degree nodes; at each step \(t\) it samples a batch \(S_t \subseteq Q\), executes it, decrements successor in-degrees, and adds newly unblocked nodes, generating a topologically ordered trajectory \(T=[S_1,\ldots,S_m]\). Arguments are grounded via the mock database to produce a verifiable Golden Trace, then an LLM converts each trace to a natural-language task. Tasks pass teacher-agent Pass@8 filtering.

Step 3 — Turn-Aware Relative Advantage (TARA). For turn \(t\) of rollout \(i\) in a group of \(K\) outputs, a binary reward \(r_{i,t}\in\{0,1\}\) checks whether the generated tool-call JSON covers the golden trace at that turn. The total advantage is: $\(A^{\text{total}}_{i,t} = A^{\text{local}}_{i,t} + \lambda \cdot A^{\text{future}}_{i,t}\)$ where the local term normalizes \(r_{i,t}\) against the group mean/std at turn \(t\), and the future term is a consistency-gated discounted sum \(V_{i,t} = \delta_{i,t}\sum_{k=0}^{T-t}\gamma^k r_{i,t+k+1}\) (gate \(\delta_{i,t}=r_{i,t}\) prevents crediting futures after a failed step), also group-normalized. Default \(\lambda=0.5\), \(\gamma=0.5\). Advantage is then uniformly propagated to all tokens within the corresponding turn.

Key Contributions

  • Automated closed-loop pipeline converting raw MCP schemas into validated executable environments; 422 diverse domains retained after testing.
  • GUST dataset: TDG + DUS generates causally coherent long-horizon tasks verified via LangGraph replay and Pass@8 teacher filtering.
  • TARA algorithm: turn-level binary rewards with consistency gating replace trajectory-level GRPO advantage, giving denser and more precise policy gradient signals.
  • Model-agnostic RL training paradigm validated across thinking and non-thinking model families (Qwen3-4B/8B, Qwen2.5-14B).

Results

  • Qwen3-8B + TARA vs. base on ACEBench-Agent: +15.15 (41.28→56.66 with GRPO, 61.66 with TARA); τ²-Bench: +8.62 (28.88→37.50).
  • Qwen2.5-14B-Instruct + TARA vs. base on ACEBench-Agent: +13.88; τ²-Bench: +6.24.
  • Qwen3-4B + TARA vs. base on ACEBench-Agent: +13.72; τ²-Bench: +2.87.
  • TARA vs. GRPO (Qwen3-8B, τ²-Bench): 32.37% vs. 30.10%; ACEBench-Agent: 61.66% vs. 56.66%.
  • Qwen2.5-7B-Instruct + TARA vs. ToolRL/agentflow/SimpleTIR on τ²-Bench: 29.87% vs. 16.37% / 17.03% / 13.53%; BFCL-v3: 20.00% vs. 15.25% / 11.25% / 11.75%.
  • Environment scaling (Qwen3-8B, TARA): 100 → 422 environments raises BFCL-v3 from 35.00% → 37.50% and τ²-Bench from 27.33% → 32.37%.
  • TARA ablation (Qwen3-8B, τ²-Bench): turn-local only 27.33%; local+future without gate 28.17%; full TARA 32.37%.

Limitations

  • SALT and FTRL cannot be directly compared due to unavailable public training artifacts; reported gap may not be apples-to-apples.
  • TDG construction relies on LLM inference and requires post-hoc sanitization (cycle removal, non-executable edge pruning), introducing a potential quality bottleneck.
  • Evaluation is confined to three existing benchmarks; generalization to entirely out-of-distribution real-world APIs is unverified.
  • Environment diversity is bounded by available open-source and proprietary MCP repositories; coverage of specialized or safety-critical domains is likely sparse.
  • The consistency gate \(\delta_{i,t}=r_{i,t}\) completely blocks future credit propagation after any failed turn, which may be overly conservative for partially correct trajectories.

Relevance to Agentic AI / LLM Agents

ToolVerse directly targets the training infrastructure gap that limits agentic RL to toy settings: by scaling executable MCP environments by two orders of magnitude and providing a principled curriculum via dependency-graph sampling, it lays groundwork for training general-purpose tool-using agents rather than specialists. The TARA credit assignment scheme addresses a fundamental training-stability problem in any multi-turn, multi-tool agent pipeline that uses GRPO-family algorithms, making it a reusable building block independent of the environment. The result that environment diversity (100→422 toolsets) monotonically improves out-of-distribution benchmark performance is a practically important scaling finding for researchers designing agentic training regimes.