What Can RL Bring to VLA Generalization? An Empirical Study

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper empirically investigates whether and how reinforcement learning (RL) fine-tuning improves out-of-distribution (OOD) generalization in Vision-Language-Action (VLA) models compared to supervised fine-tuning (SFT). Using a new benchmark with three generalization axes (Vision, Semantics, Execution) on pick-and-place tasks, the authors find PPO yields strong gains in semantic and execution robustness while matching SFT on visual perturbations. They also distill an efficient PPO recipe for VLAs based on a shared actor-critic backbone, warm-up initialization, and single-epoch updates.

Problem

SFT-trained VLAs suffer from compounding errors under distribution shift because behavioral cloning only mimics expert demonstrations, leaving policies brittle to novel objects, instructions, robot poses, and mid-episode perturbations. A systematic, dimension-wise understanding of where RL fine-tuning helps VLAs—versus merely repeating prior LLM findings—was absent.

Method

The study bases all experiments on OpenVLA (SigLIP + DINOv2 visual encoder + LLaMA-2 7B backbone), fine-tuned with LoRA (rank 32) in a ManiSkill simulation environment with an 8-DoF WidowX-250S arm.

RL algorithm selection: PPO, GRPO, and iterative DPO are evaluated. PPO uses clipped policy gradients with Generalized Advantage Estimation (GAE); a variant (PPO-ORZ) disables GAE (γ=λ=1). GRPO samples 8 trajectories per initial state for group-relative advantage. DPO infers pairwise preferences from sparse rewards offline. PPO consistently dominates; GRPO struggles due to POMDP non-stationarity destabilizing group advantage estimates, and DPO fails due to sparse rewards and offline-to-online distribution shift.

PPO recipe: (1) Shared actor-critic backbone—a 3-layer MLP value head attaches to the first action-token hidden state h₀ of the final transformer block, saving 83% VRAM vs. a separate critic. (2) SFT warm-up on 140 motion-planner demonstrations halves convergence steps. (3) Minimal PPO epoch (epoch=1) achieves peak sample efficiency; more epochs yield no gain but increase wall-clock time linearly. Full training requires ~42 hours on a single A100.

Benchmark: Three axes—Vision (dynamic textures, unseen tables, image noise at weak/strong levels), Semantics (unseen objects, receptacles, instruction phrasings, multi-object, distractive receptacle), Execution (unseen object/receptacle positions, unseen robot init pose, mid-episode object reposition). 9 novel objects, 16 unseen receptacles, 5 new tables, 16 distractor textures held out at test time. Assets drawn from Objaverse; tables synthesized with Stable Diffusion + ControlNet.

Key Contributions

• A structured, three-axis OOD generalization benchmark for VLAs covering Vision, Semantics, and Execution dimensions.
• Identification of PPO as the preferred RL algorithm for VLA fine-tuning; demonstration that LLM-derived methods (GRPO, DPO) fail due to POMDP dynamics and sparse rewards.
• An efficient PPO training recipe: shared actor-critic with h₀ critic input, SFT warm-up, epoch=1, converging in ~42 GPU-hours on one A100.
• Empirical finding: RL surpasses SFT by 42.6% on OOD objects and tables at convergence; achieves superior execution robustness (recovery from failed grasps, mid-episode reposition adaptation) while matching SFT on visual robustness.

Results

• RL vs. SFT overall OOD (unseen objects + tables): RL outperforms SFT-16k by 42.6% at convergence; RL overtakes SFT-16k after ~0.4M environment steps.
• Vision: RL ≈ SFT—neither method generalizes beyond training-time visual randomization.
• Semantics: RL noticeably better—stronger on unseen single and multi-object scenarios; improved instruction-phrasing generalization.
• Execution: RL significantly better across all three sub-tasks (OOD object/receptacle position, OOD robot init pose, mid-episode object reposition); SFT stalls or fails when distribution shifts involve control dynamics.
• PPO vs. GRPO/DPO: PPO reaches success rate ~0.65–1.0; GRPO and DPO show near-zero or stagnant improvement across 1M environment steps.
• Shared vs. separate critic: Separate critic achieves comparable reward but trains 35% slower and uses 83% more VRAM (81.3 GB vs. 44.4 GB); h₀ input outperforms hn and [h₀,…,hn] concatenation.
• SFT data scaling: Performance plateaus at 16k trajectories both in-distribution and OOD; 64k trajectories yield no further gains.

Limitations

• Demonstrations for SFT warm-up are generated by a motion planner (Octo-Small + MPLib), not human teleoperators, potentially underestimating SFT's ceiling with higher-quality data.
• Evaluation is restricted to a single task class (pick-and-place); generalization to multi-task or long-horizon manipulation is untested.
• All experiments are in simulation (ManiSkill); sim-to-real transfer validation on physical robots is left as future work.
• Visual robustness is not improved by RL over SFT, suggesting RL's benefits do not extend to novel visual domains beyond the training randomization.
• Study uses a single VLA backbone (OpenVLA/LLaMA-2 7B); findings may not transfer to architectures with different action representations (diffusion, flow matching).

Relevance to Vision-Language Models

This paper directly addresses a central question in VLM-to-VLA transfer: whether RL post-training techniques validated for LLMs/VLMs (PPO, GRPO, DPO) carry over to the embodied action-generation setting. The finding that GRPO and DPO fail for VLAs—while effective for VLMs—highlights a fundamental gap between token-generation tasks and continuous POMDP control, informing how the VLM community should adapt reasoning-RL recipes for robotics. The semantic generalization gains from PPO parallel the "SFT memorizes, RL generalizes" result observed for language models, extending that principle to multimodal grounding and physical manipulation. For VLM researchers, the efficient shared actor-critic recipe and dimension-wise OOD decomposition offer a reusable methodology for evaluating fine-tuning strategies in embodied settings.