Skip to content

SimpleVLA-RL: Scaling VLA Training via Reinforcement Learning

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; SimpleVLA-RL: scaling VLA training via reinforcement learning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SimpleVLA-RL applies outcome-driven reinforcement learning (binary success/failure reward) to Vision-Language-Action (VLA) models, extending the DeepSeek-R1/GRPO paradigm from text reasoning to closed-loop robotic manipulation. It achieves 99.1% average success on LIBERO and 80% relative improvement on RoboTwin 2.0 over a strong SFT baseline, while dramatically reducing dependence on demonstration data. A surprising emergent behavior, "pushcut," shows RL discovering action patterns never present in training trajectories.

Problem

VLA models trained via supervised fine-tuning (SFT) face two compounding bottlenecks: (1) data scarcity—expert robotic trajectories are expensive and rare, constraining scale; and (2) poor generalization—models memorize scene-specific patterns and fail under distribution shift (unseen objects, spatial configurations, long-horizon compositions). Traditional RL alternatives require hand-crafted per-task reward functions, limiting scalability. Existing RL frameworks also lack VLA-specific infrastructure for closed-loop environment interaction and parallelized rendering.

Method

SimpleVLA-RL builds on the veRL framework with four key VLA-specific adaptations:

Trajectory rollout: Unlike LLMs (open-loop token generation), VLA rollout is closed-loop—each action \(A_t\) updates the environment state \(S_{t+1}\), requiring continuous visual feedback. The policy (OpenVLA-OFT, LLaMA2-7B backbone with token-based action decoding) samples \(G\) trajectories per query via temperature-scaled random sampling over action token distributions.

Outcome reward: Binary trajectory-level reward \(R(a_{i,t} | s_{i,t}) \in \{0,1\}\) based solely on task completion, uniformly propagated to all action tokens within each trajectory. No process-level or task-specific reward engineering.

GRPO training objective (modified): $\(J(\theta) = \mathbb{E}\left[\frac{1}{G}\sum_{i=1}^G \frac{1}{|a_i|}\sum_{t=1}^{|a_i|} \min\left(r_{i,t}(\theta)\hat{A}_i,\ \text{clip}(r_{i,t}(\theta), 1-\varepsilon_L, 1+\varepsilon_H)\hat{A}_i\right)\right]\)$ with group-normalized advantage \(\hat{A}_i = (R_i - \text{mean})/\text{std}\). KL divergence regularization is removed (no reference model needed).

Three exploration enhancements: 1. Dynamic Sampling: Reject rollout batches where all \(G\) trajectories share identical outcomes (all-succeed or all-fail), enforcing \(0 < |\text{successes}| < G\) to guarantee non-zero advantage estimates. 2. Clip-Higher: Asymmetric clip range \([0.8, 1.28]\) instead of symmetric \([0.8, 1.2]\), loosening the upper bound to allow low-probability action tokens to increase in probability. 3. Higher Rollout Temperature: Sampling temperature raised from 1.0 to 1.6 during rollout to diversify trajectory exploration.

Infrastructure: veRL extended with parallel multi-environment rendering and a unified train–inference–render pipeline for distributed VLA RL.

Key Contributions

  • End-to-end online RL framework for token-based VLA models with VLA-specific rollout, binary outcome reward, and scalable parallel rendering
  • Three exploration-enhancing strategies (dynamic sampling, clip-higher, higher temperature) each contributing 10–15% performance gains
  • Demonstrated data efficiency: One-Trajectory SFT (10 demos per suite) + RL reaches 96.9% average success on LIBERO, surpassing Full-Trajectory SFT (91.0%) with 500 demos per suite
  • State-of-the-art results on LIBERO, RoboTwin 1.0, and RoboTwin 2.0; strong sim-to-real transfer without real-robot data
  • Discovery of the "pushcut" phenomenon: RL training elicits novel manipulation strategies (e.g., push-then-cut actions) absent from any supervised demonstration data

Results

LIBERO (language-guided manipulation, 4 suites): - SimpleVLA-RL: 99.1% avg (Spatial 99.4, Object 99.1, Goal 99.2, Long 98.5) vs. OpenVLA-OFT SFT baseline 91.0%; surpasses π0 (94.2%) and UniVLA (95.2%) - One-Trajectory SFT + RL: 96.9% avg vs. One-Trajectory SFT 48.9% (LIBERO-Long: 17.3% → 91.7%, +74.4 pp)

RoboTwin 1.0 (4 bimanual tasks): - OpenVLA-OFT SFT: 39.8% → 70.4% with RL (+30.6 pp); beats DP3 (58.1%)

RoboTwin 2.0 (12 tasks, domain-randomized, 3 horizon levels): - OpenVLA-OFT SFT: 38.3% → 68.8% (+30.5 pp, ~80% relative improvement); surpasses π0 (52.7%) and RDT (33.3%) - Short-horizon avg: 21.3% → 64.9%; Long/Extra-Long avg: 46.5% → 69.0%

Generalization (unseen tasks, LIBERO): - RL consistently improves unseen task SR while SFT catastrophically forgets (drops to 0% on Goal unseen tasks); RL gains 5–36.5% on held-out tasks across all suites

Real-world (sim-to-real, 4 tasks): - OpenVLA-OFT SFT: 38.0% avg → 70.0% avg with RL on Stack Bowls; +32 pp over SFT, outperforming RDT (23.5% avg)

Limitations

  • Requires a pre-trained SFT base with non-zero success rate before RL can be applied—zero-shot RL from scratch on LIBERO fails (0% baseline)
  • Evaluated on a single backbone (OpenVLA-OFT/LLaMA2-7B); generality to diffusion-based or MLP-regression VLAs is unverified, as those are incompatible with policy-gradient RL
  • Binary outcome reward provides no intermediate learning signal, which may be insufficient for extremely sparse or partially-rewarded long-horizon tasks beyond those tested
  • Sim-to-real results are incomplete in the provided text (Place Empty Cup RL result is cut off); real-world evaluation covers only 4 tasks
  • "Pushcut" phenomenon is observed but not mechanistically explained; conditions for novel pattern emergence are unclear

Relevance to Vision-Language Models

SimpleVLA-RL directly addresses a central challenge in embodied VLMs: overcoming the data bottleneck that limits SFT-based scaling by replacing static imitation with online RL. The paper draws an explicit analogy to DeepSeek-R1's finding that sparse outcome rewards unlock step-by-step reasoning in LLMs, arguing and demonstrating this transfers to step-by-step action planning in VLAs—a key theoretical bridge for the VLM community. The "pushcut" emergent behavior is particularly relevant: it suggests that RL on VLMs can produce genuinely novel compositional behaviors, not merely interpolation over training data, which has implications for how researchers think about grounding, generalization, and skill emergence in multimodal foundation models. For those tracking VLMs in robotic contexts, this is a strong empirical result that RL post-training (mirroring RLHF/RLAIF patterns in language models) may be the next scaling axis for embodied VLMs.