On Effectiveness and Efficiency of Agentic Tool-calling and RL Training

🕒 Published (v1): 2026-05-28 22:21 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML; systematic study of tool-calling effectiveness and RL training tradeoffs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper audits agentic tool-calling along two axes: evaluation reliability and RL training efficiency. It shows that minor, undocumented choices in evaluation pipelines can swing BFCL benchmark scores substantially, and it proposes two lightweight techniques that accelerate GRPO-based RL training by 1.7–2.6× without performance loss.

Problem

Tool-calling benchmarks and RL training pipelines for LLM agents lack rigorous standardization. On the evaluation side, implementation choices (random seed, multi-turn template, thinking-history retention, system prompt) cause large, unreproducible performance swings that distort leaderboard comparisons. On the training side, standard RL algorithms (PPO, GRPO) waste computation: up to 80% of prompts yield zero gradient signal during rollouts, and the policy-update stage costs 3–5× more wall-clock time than rollout generation.

Method

Effectiveness audit: Using BFCL as a case study, the authors systematically vary four evaluation axes across five models (Qwen3-4B/8B, Qwen2.5-7B, Llama3.1-8B, Llama3.2-3B): - Random seeds: 10-seed sweeps to quantify variance. - Multi-turn template: "native" role–content serialization vs. "context" (full history injected as a single user turn). - Thinking history: with vs. without <think> block retention across turns. - System prompt: small manual additions to the default BFCL prompt.

Efficiency interventions (on top of GRPO):

1. Online pre-rollout filtering: Maintain a per-prompt streak counter \(c_{i,k+1}^{(e)}\) tracking consecutive epochs in which all \(n\) rollouts achieved maximum reward. Skip prompt \(s_{i,k}\) at epoch \(e\) if \(c_{i,k+1}^{(e-1)} \geq k\). The active dataset shrinks as $\(D^{(e)} = \{s_{i,k+1} \in D^{(e-1)} : c_{i,k+1}^{(e)} < k\},\)$ eliminating redundant rollouts on already-solved prompts.

2. Max-variance rollout down-sampling: Generate \(n\) rollouts but backpropagate through only \(m < n\) rollouts chosen to maximize reward variance. The optimal subset \(S^* = \{1,\ldots,m'\} \cup \{n-(m-m')+1,\ldots,n\}\) selects the \(m'\) lowest- and \((m-m')\) highest-reward rollouts, reducing policy-update cost by \(\approx n/m\).

Training uses VERL; single-turn on Qwen2.5-3B-Instruct (xLAM + ToolACE, 2.3k examples); multi-turn on Qwen3-4B (curated 2.6k trajectories).

Key Contributions

• Systematic empirical characterization of four evaluation sensitivity axes in BFCL multi-turn tool-calling.
• Quantification that up to 80% of GRPO training prompts are zero-variance (no gradient signal) during tool-calling RL.
• Identification of a 3–5× computation asymmetry between policy-update and rollout stages in tool-calling vs. math RL.
• Online pre-rollout filtering that exploits temporal stability of all-correct prompts (\(P(\text{still all-correct} \mid k\text{-consecutive all-correct}) > 0.8\)).
• Max-variance rollout down-sampling that cuts update cost by \(n/m\).
• 1.7× (single-turn) and 2.6× (multi-turn) end-to-end wall-clock speedup at matched final performance.
• Finding that multi-turn training data does not reliably improve multi-turn BFCL; single-turn data preserves multi-turn performance.

Results

• Seed variance: Single-turn BFCL is stable; multi-turn deviates up to ~3% across seeds.
• Template: Native serialization outperforms context concatenation by ~6–8% on BFCL multi-turn.
• Thinking history: Retaining <think> blocks improves Qwen3 models by ~3–5% on BFCL multi-turn.
• System prompt: A small manual prompt change for Qwen3-4B yields multi-turn gains comparable to or exceeding full RL fine-tuning.
• Training format (Qwen3-4B): Single-turn training: BFCL multi-turn 20.2 ± 0.6, non-live 89.5 ± 0.5, live 79.9 ± 0.2. Multi-turn training: 15.9 ± 0.4, 88.6 ± 0.4, 80.7 ± 0.1 — multi-turn supervision degrades multi-turn BFCL.
• Efficiency: GRPO + both techniques achieves 1.7× speedup (single-turn) and 2.6× speedup (multi-turn) at matched performance vs. vanilla GRPO.
• ACEBench generalization: RL-trained Qwen3-4B improves +12.1 overall accuracy points over base model, outperforming Nova1-Lite and open-source baselines (Gemma3-27B, Magistral-Small-24B).
• Baselines: Claude Sonnet 4, Nova 1 Lite/Pro (closed-source); Gemma3-Instruct-27B, Magistral-Small-2509-24B (open-source).

Limitations

• Effectiveness analysis is confined to BFCL; sensitivity patterns may not fully generalize to other benchmarks (ACEBench is used for partial transfer validation only).
• Multi-turn training data quality bottleneck is identified but not solved; the paper does not propose a method for higher-quality multi-turn trajectory curation.
• Online pre-rollout filtering relies on a fixed threshold \(k\) (empirically \(k=1\) or \(2\)); dynamic or per-prompt scheduling is not explored.
• Max-variance down-sampling was originally proposed by Xu et al. (2025) for math; the paper adapts it to tool-calling but does not claim algorithmic novelty for this component.
• Training is restricted to small-to-mid-sized models (≤8B parameters); generalization to larger models is noted but not fully demonstrated.
• Conflict of interest: authors are Amazon employees, and Nova-1 (an Amazon model) is one of the evaluated baselines.

Relevance to Agentic AI / LLM Agents

This paper directly addresses two foundational concerns for anyone building or evaluating LLM agents with tool use: whether benchmark numbers are trustworthy and whether RL training is computationally tractable. The finding that system-prompt choices alone can match RL fine-tuning gains in multi-turn tool-calling is a sharp warning for the community that current leaderboards may not reflect genuine capability progress. The efficiency techniques are practically significant for research groups training tool-calling agents with GRPO/PPO, where the 3–5× update-vs-rollout asymmetry is a previously underreported bottleneck. Together, these contributions push toward more reproducible evaluation standards and more efficient post-training pipelines for agentic systems.