Skip to content

When Does Muon Help Agentic Reinforcement Learning?

🕒 Published (v1): 2026-07-17 17:49 UTC · Source: Arxiv · link

Why this paper was selected

Muon vs AdamW in sparse-reward agentic RL; optimizer choice has direct post-training implications

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper empirically investigates whether the Muon optimizer (spectral normalization via Newton–Schulz iterations) benefits LLM agents trained with sparse-reward RL post-training. Using matched single-seed comparisons on ALFWorld with Qwen2.5-0.5B-Instruct, they find that Muon applied only to hidden weight matrices yields an 88% relative gain in final-window validation success under GiGPO, but the benefit depends critically on the advantage estimator and learning rate.

Problem

Muon is well-established for pre-training efficiency but its utility for RL post-training was an open question, with prior work reporting failures in single-turn RLVR (outcome-only reward) settings. The mechanism behind those failures—attributed to spectral whitening amplifying noise in low-SNR policy gradients—suggests that the structure of the advantage estimator may determine whether Muon helps. No prior work examined Muon across multiple credit-assignment designs in long-horizon, multi-turn agentic RL.

Method

All experiments run on ALFWorld (six household task categories, horizon ≤50 steps, sparse terminal reward) training Qwen2.5-0.5B-Instruct for 200 updates on eight H20 GPUs using verl-agent. Muon is applied only to 2D hidden weight matrices (all attention and MLP weights); embeddings, layer norms, and the tied LM head use AdamW at \(\eta = 10^{-6}\) in every configuration. Three advantage estimators are compared:

  • GRPO: episode-level normalized returns \(A^E(\tau_i) = \frac{R(\tau_i) - \mu}{\sigma}\)
  • GiGPO: \(A(a_t^{(i)}) = A^E(\tau_i) + \omega A^S(a_t^{(i)})\), where \(A^S\) is a step-level advantage from grouping actions at recurring anchor states; setting \(\omega=0\) recovers GRPO in the same code path
  • GraphGPO: transition-level advantages via a unified state-transition graph with goal-distance assignment

Muon learning rates tested: \(3\times10^{-5}\) and (for GiGPO/GraphGPO) \(10^{-5}\). AdamW controls at \(10^{-5}\) and \(3\times10^{-5}\) test whether higher nominal AdamW rates account for the gain. Primary metrics: mean validation success over steps 175–200 ("late-window") and normalized validation AUC (trapezoidal area under the full 0–200 step curve, divided by 200). A 2×2 \(\{\)AdamW, Muon\(\} \times \{\omega=0, \omega=1\}\) factorial isolates the step-credit vs. optimizer contributions.

Key Contributions

  • Controlled single-seed demonstration that vanilla Muon improves long-horizon sparse-reward agentic RL: GiGPO + Muon raises final-window success from 0.290 to 0.546 (+88% relative) and from 0.32 to 0.63 at the final checkpoint.
  • Learning-rate controls showing the gain is not replicated by increasing AdamW's nominal rate: AdamW at \(10^{-5}\) and \(3\times10^{-5}\) both reach zero post-update validation success, while Muon at \(3\times10^{-5}\) retains success through all 200 steps.
  • Matched 2Ă—2 GiGPO factorial (\(\omega \in \{0,1\}\)) showing Muon improves at both step-credit settings (+0.220 at \(\omega=0\), +0.255 at \(\omega=1\)) and that step-level credit and Muon are complementary.
  • A gradient-SNR conjecture: Muon's polar-factor update \(\Delta W \propto UV^\top\) is useful only when weak singular directions carry reliable signal; finer credit (GiGPO anchor-state contrasts, GraphGPO transition-level assignment) may raise some \(s_i/\sigma_i\) ratios and explain the estimator-dependent pattern.

Results

  • GiGPO, Muon \(3\times10^{-5}\): late-window 0.546 vs. AdamW 0.290 (\(\Delta = +0.255\)); \(\Delta\text{AUC} = +0.127\)
  • GiGPO, Muon \(10^{-5}\): late-window 0.509; \(\Delta\text{AUC} = +0.147\) (slightly higher AUC than \(3\times10^{-5}\))
  • GRPO, Muon \(3\times10^{-5}\): late-window 0.268 vs. 0.161 (\(\Delta = +0.107\)); \(\Delta\text{AUC} = +0.013\) (gain concentrated late)
  • GraphGPO, Muon \(10^{-5}\): late-window 0.901 vs. 0.810 (\(\Delta = +0.091\)); normalized AUC 0.556 vs. 0.399 (\(\Delta = +0.157\)); reaches 0.5/0.75 success thresholds ~30/60 steps earlier than AdamW
  • GraphGPO, Muon \(3\times10^{-5}\): late-window 0.828 vs. 0.810; \(\Delta\text{AUC} = +0.076\) (late-window narrows near saturation)
  • AdamW controls (GiGPO): both \(10^{-5}\) and \(3\times10^{-5}\) collapse to zero validation success from step 5 onward
  • 2Ă—2 ablation: disabling step credit (\(\omega=0\)) drops AdamW to 0.141 and Muon to 0.361; enabling it (\(\omega=1\)) improves both to 0.290 and 0.546 respectively

Limitations

  • Single-seed throughout; reported values describe individual trajectories, not population-level statistics.
  • Single model (Qwen2.5-0.5B-Instruct) and single environment (ALFWorld); cross-task and cross-model generalization unverified.
  • Narrow learning-rate sweep: GRPO tested only at \(3\times10^{-5}\); AdamW controls only under GiGPO.
  • The gradient-SNR conjecture is unverified—no direct measurement of update spectra, effective rank, or per-direction SNR.
  • Full-matrix NO_SHARD Newton–Schulz requires unsharded 2D matrices, increasing per-device memory and precluding straightforward use with larger models without distributed Muon variants.

Relevance to Agentic AI / LLM Agents

Optimizer choice is an under-studied infrastructure variable for training LLM agents via RL, and this paper provides the first controlled evidence that Muon can substantially outperform AdamW in long-horizon, multi-turn agentic RL—a regime that differs from the single-turn RLVR setting where Muon previously failed. The finding that advantage estimator design (GRPO vs. GiGPO vs. GraphGPO) mediates whether Muon helps means that practitioners training agents with group-based RL algorithms cannot treat the optimizer and credit-assignment method as independent choices. The gradient-SNR framing also suggests a principled direction for future work: designing advantage estimators that improve signal quality in the policy-gradient spectrum, which could unlock spectral optimizers more broadly.