Skip to content

UP: Unbounded Positive Asymmetric Optimization for Breaking the Exploration-Stability Dilemma

🕒 Published (v1): 2026-07-08 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

UP (Unbounded Positive Asymmetric Optimization) resolves the exploration-stability dilemma in importance-sampling-based RL for LLM reasoning by replacing the historical policy anchor \(\pi_{\text{old}}\) with a stop-gradient self-anchor \(\text{sg}(\pi_\theta)\) exclusively for correct rollouts, yielding an unclipped gradient mathematically equivalent to REINFORCE. The method is a plug-and-play objective that integrates with GRPO, DAPO, and GSPO at both token and sequence granularities, and is validated on dense, MoE, and vision-language model architectures.

Problem

Standard IS-based RL algorithms (GRPO, DAPO, GSPO) face a structural dilemma: unclipped importance sampling causes gradient explosion on rare low-probability trajectories, while the clipping mechanism used to prevent this strictly upper-bounds the policy update to \(\pi_\theta \leq (1+\epsilon_{\text{high}})\pi_{\text{old}}\), severely limiting the update budget for low-confidence correct tokens. For example, with \(\pi_{\text{old}} = 0.01\) and \(\epsilon_{\text{high}} = 0.28\), the maximum absolute probability increase is only \(0.0028\)—structurally preventing reinforcement of rare but correct reasoning paths.

Method

UP introduces the Probability Capacity (Cap) concept to formalize the update budget constraint. For positive-advantage tokens, UP replaces the IS ratio \(r_{i,t}(\theta) = \pi_\theta / \pi_{\text{old}}\) with a self-anchored ratio:

\[\tilde{r}_{i,t}(\theta) = \frac{\pi_\theta(o_{i,t}|q, o_{i,<t})}{\text{sg}(\pi_\theta(o_{i,t}|q, o_{i,<t}))}\]

where \(\text{sg}(\cdot)\) denotes the stop-gradient operator. Because the denominator is a constant during backprop, the gradient reduces exactly to the REINFORCE gradient \(\mathbb{E}[\hat{A}\nabla_\theta \log \pi_\theta]\)—stable and unclipped. For negative-advantage tokens, standard DAPO/GRPO clipping is retained to prevent instability from aggressive penalization. This asymmetric routing (unbounded for \(\hat{A}>0\), clipped for \(\hat{A}\leq 0\)) is instantiated as UP-DAPO (token-level), UP-GRPO (token-level with KL penalty), and UP-GSPO (sequence-level using the length-normalized ratio \(s_i(\theta)\)).

Key Contributions

  • Formal definition of Probability Capacity (Cap) exposing that clipping linearly constrains updates to \(\min(1, (1+\epsilon_{\text{high}})\pi_{\text{old}}) - \pi_\theta\), stifling exploration for low-confidence tokens.
  • Theoretical proof that replacing \(\pi_{\text{old}}\) with \(\text{sg}(\pi_\theta)\) as the IS denominator yields an objective whose gradient is identically the REINFORCE gradient, eliminating IS-induced instability without clipping.
  • Universal asymmetric objective (UP) that plugs into GRPO, DAPO, and GSPO across token-level and sequence-level granularities.
  • Validation across dense (Qwen3-14B-Base, Qwen3-8B), MoE (Qwen3-30B-A3B-Base), and vision-language (Qwen3-VL-8B-Instruct) architectures.

Results

  • UP-DAPO vs. DAPO on AIME24 (Qwen3-14B-Base): Avg@32 51.15 vs. 47.71; Maj@32 60.88 vs. 58.36; Best@32 81.79 vs. 80.49.
  • UP-DAPO achieves higher generation entropy throughout training while maintaining comparable or lower gradient norm and KL divergence vs. DAPO baseline.
  • UP-GRPO vs. eleven baselines (GRPO, Dr. GRPO, CISPO, DPPO, GMPO, GSPO, SAPO, REINFORCE++, RLOO, W-REINFORCE, ASPO): best average Pass@1 across five benchmarks—AIME24, AMC23, MATH500, Minerva, OlympiadBench.

Limitations

  • The unbounded positive gradient is mathematically REINFORCE; on-policy variance concerns (known weakness of REINFORCE) are not fully analyzed in the multi-step IS setting.
  • Multimodal (VLM) results are reported only on Geometry3K; broader multimodal reasoning benchmarks are not evaluated.
  • The method still depends on DAPO/GRPO clipping for the negative branch; behavior under reward noise or mis-specified advantages is not characterized.
  • Paper text is truncated; ablation studies (Sec. 5.3 onward) and full baseline comparisons across all model architectures are not available for review.

Relevance to Vision-Language Models

UP is validated on Qwen3-VL-8B-Instruct (trained on Geometry3K), demonstrating that asymmetric RL objectives transfer to the multimodal setting where RL-based reasoning training is increasingly common for VLMs. The exploration bottleneck UP addresses is particularly acute in VLMs, where correct visual-reasoning chains are sparse and low-probability under the initial policy, making conservative clipping especially harmful. As RL fine-tuning of VLMs (e.g., for math, chart, and diagram reasoning) gains traction, UP provides a plug-in improvement to any GRPO/DAPO-style training pipeline already deployed in the VLM community.