Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learning

🕒 Published (v1): 2026-07-08 15:02 UTC · Source: Arxiv · link

Why this paper was selected

Jie Tang/Yuxiao Dong group; async RL eliminates synchronization bottleneck for long-horizon agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SAO (Single-rollout Asynchronous Optimization) replaces group-wise sampling (GRPO) with single-rollout updates and a value-based critic to make asynchronous RL training of LLMs stable at scale. It introduces strict double-sided token-level importance-sampling clipping and a skip-observation GAE for agentic trajectories. SAO trains stably for 1,000 steps and consistently outperforms GRPO on agentic coding and math reasoning benchmarks.

Problem

Synchronous RL pipelines idle GPUs waiting for the slowest rollout in a batch; asynchronous RL mitigates this but introduces two compounding problems: (1) policy lag — trajectories may be generated by arbitrarily stale model versions, making off-policy correction unpredictable; and (2) GRPO's group-wise sampling is fundamentally mismatched to async settings because the group must wait for its slowest member before any update can proceed, exacerbating the policy-lag problem and making online/single-feedback environments (where only one trajectory per prompt is available) intractable.

Method

SAO combines four mechanisms:

Direct Double-Sided Importance Sampling (DIS). Rather than tracking a separate "old policy" \(\pi_{\theta_\text{old}}\), SAO uses the rollout engine's log-probabilities directly as the behavior proxy: \(r_t(\theta) = \exp(\log\pi_\theta(a_t|s_t) - \log\pi_\text{rollout}(a_t|s_t))\). Tokens outside the trust region \([1-\epsilon_\ell, 1+\epsilon_h]\) are masked from gradient computation entirely (rather than merely clipped), giving the objective \(L(\theta) = \hat{\mathbb{E}}_t[f(r_t(\theta),\epsilon_\ell,\epsilon_h)\hat{A}_t \log\pi_\theta(a_t|s_t)]\) where \(f=r_t\) inside the interval and \(f=0\) outside.

Single-rollout sampling. One trajectory per prompt replaces GRPO's group of \(N\) responses, eliminating group-wait latency. Advantage estimation then requires a reliable value model.

Value-model stabilization. Two strategies: (a) Faster value update — \(K=2\) critic gradient steps per policy step to keep advantage estimates aligned with the current policy; (b) Frozen-attention — attention parameters of the value model are frozen during RL, only MoE projection weights are updated, which suppresses large critic gradient norms observed in full-parameter training.

Skip-observation GAE. For agentic trajectories \([a_0, o_0, a_1, o_1, \ldots]\), the standard GAE boundary crossing from action to observation introduces noise because the model does not generate \(o_i\). SAO bridges this gap: $\(\hat{A}(a_{i,N}) = \delta + \gamma\lambda\hat{A}(a_{i+1,0}), \quad \delta = r_t + \gamma V(a_{i+1,0}) - V(a_{i,N})\)$ thus constraining advantage estimation to model-generated tokens only.

Key Contributions

• Drop-in replacement of group-wise sampling with single-rollout updates, enabling true async RL without group-wait stalls.
• DIS: simplified importance sampling using rollout log-probs directly plus strict bilateral masking that eliminates the need to maintain a checkpoint history of old policies.
• Skip-observation token-level GAE that handles interleaved environment feedback in agentic trajectories.
• Empirical recipe for value-model stabilization (frozen attention + TTUR \(K=2\)), supported by explained-variance and gradient-norm diagnostics.
• Online-learning simulation showing SAO's critic can track non-stationary reward distributions, unlike running-mean baselines.
• Deployment at scale: SAO used in the production RL pipeline for the open GLM-5.2 (750B-A40B) model.

Results

All reasoning benchmarks use Qwen3-30B-A3B as the SFT baseline (Pass@1, with Python tool); SWE-Bench uses the same base model with OpenHands scaffold.

• AIME2025: SAO 97.3% vs. GRPO 84.2% vs. SFT baseline 80.4%
• BeyondAIME: SAO 74.8% vs. GRPO 54.8% vs. SFT baseline 53.3%
• HMMT Nov 2025: SAO 88.3% vs. GRPO 76.0% vs. SFT baseline 75.2%
• IMOAnswerBench: SAO 74.0% vs. GRPO 55.8% vs. SFT baseline 53.3%
• SWE-Bench Verified: SAO 29.8% vs. GRPO+DIS 27.0% vs. baseline 23.0%
• Vanilla GRPO collapses at ~160 training steps; GRPO+DIS stabilizes but diverges from SAO after ~400 steps; SAO remains stable at 1,000 steps.
• Ablation: removing faster value update drops AIME to 95.0%; removing frozen attention drops to 90.6%; running-mean baseline (no value model) drops BeyondAIME to 55.3%.
• Online learning: SAO recovers target style reward faster than running-mean baseline after each of three reward-distribution shifts.

Limitations

• Frozen-attention strategy is motivated by MoE architecture instability (Qwen3 models); applicability to dense transformer value models is not demonstrated.
• Skip-observation GAE benefit over step-level GAE is asserted but the ablation result is deferred to an appendix not included in the provided text.
• Online learning simulation uses a synthetic stylistic-preference task; transfer to real-world non-stationary agentic environments is not validated.
• Value pretraining scale is identified as critical, but the paper does not quantify the sensitivity or specify corpus size, leaving a key hyperparameter underspecified for replication.
• The extreme asymmetry of clipping bounds (\(\epsilon_\ell = 0.3, \epsilon_h = 5.0\) for reasoning; \(\epsilon_\ell = 0.8, \epsilon_h = 3.0\) for coding) suggests task-sensitive tuning is required.
• Experiments are limited to one model family (Qwen3-30B-A3B); generality across architectures is untested.

Relevance to Agentic AI / LLM Agents

SAO directly targets the bottleneck of training LLM agents with RL on long-horizon tasks — the inefficiency and instability of synchronous group-sampling pipelines when trajectories are variable-length and environment feedback is interleaved. The skip-observation GAE and single-rollout design are specific engineering contributions for agentic (multi-turn, tool-using) settings rather than simple chat or math tasks. This work connects to the broader effort to make PPO-style actor-critic training competitive with GRPO for agentic post-training, particularly relevant to coding agents (SWE-Bench) and tool-integrated reasoning. Its deployment in GLM-5.2 at 750B scale provides an existence proof that these techniques transfer to frontier model training.