Skip to content

Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Robot-R1 applies DeepSeek-R1-style reinforcement learning (GRPO) to train LVLMs for embodied reasoning in robot control, replacing brittle SFT pipelines. It reformulates continuous next-keypoint-state prediction as multiple-choice QA to make RL exploration tractable. A 7B model trained this way outperforms GPT-4o on low-level control reasoning tasks and generalizes to out-of-distribution embodied benchmarks where SFT variants fail entirely.

Problem

SFT-based embodied reasoning datasets for robotics are heuristically constructed and not explicitly optimized for actual robot action prediction—linguistic labels often miss precise quantitative details needed for low-level control. Additionally, SFT causes catastrophic forgetting of general conversational and reasoning abilities, and models trained on fixed SFT formats fail to generalize to unseen input distributions.

Method

Robot-R1 trains Qwen2.5-7B-VL-Instruct via GRPO on three multiple-choice QA (MCQA) tasks derived from RLBench expert demonstrations: 1. Waypoint prediction QA: given current image observation and state \((o_t, s_t)\) plus environment metadata \(M\) (reference points, axis conventions, end-effector dimensions), predict next keypoint state \(s_{k^*}\) from four options (one correct, three randomly sampled distractors). 2. Current state prediction QA: identify the robot's current 3D Cartesian state from visual observation alone. 3. Movement prediction QA: predict rule-based directional label (e.g., "move up", "slightly move backward") derived from the signed difference \(s_{k^*} - s_t\).

The MCQA formulation discretizes the continuous 3D state space, making RL exploration feasible. Models output <think>...</think><answer>...</answer> chains; the GRPO reward is the sum of a format reward \(r_f\) and an exact-match answer correctness reward \(r_a\). Training uses ~7.5K QA pairs across five RLBench tasks (50 demos each), batch size 128, 5 epochs, 5 rollouts per prompt.

Key Contributions

  • Robot-R1 framework: RL-optimized embodied reasoning for LVLMs using GRPO on MCQA tasks derived from expert demonstrations.
  • MCQA reformulation of state prediction: converts continuous 3D waypoint prediction into a discrete 4-option selection problem, enabling efficient RL exploration.
  • Three-task auxiliary structure: waypoint + current-state + movement prediction QA progressively improve performance (ablation Table 9 confirms each auxiliary task adds value).
  • Robot-R1 Bench: 215 open-ended human-authored QA pairs over 10 RLBench tasks, covering planning, high-level action, movement, and spatial reasoning, evaluated with GPT-4o-as-judge (Pearson r ≈ 0.9 vs. human for all but planning).

Results

  • Robot-R1 Bench (low-level control): ROBOT-R1 achieves movement score 0.76 and spatial score 1.51, surpassing GPT-4o (0.72 / 1.43) and all evaluated commercial models; Direct SFT collapses to near-zero (0.06 / 0.06).
  • Robot-R1 Bench (high-level control): High-level action score improves from 1.04 (base) to 1.30 (ROBOT-R1); planning slightly regresses from 1.66 to 1.44–1.70 across seeds.
  • EmbodiedBench Manipulation: avg success rate 8.92% → 11.68% (~31% relative gain); both Direct SFT and CoT SFT achieve 0% on all tasks.
  • SpatialRGPT-Bench: qualitative accuracy 29.07% → 40.79% (~40% gain); quantitative accuracy 9.88% → 15.89% (~60% gain); SFT variants degrade below baseline on most sub-tasks.
  • RL algorithm ablation: RLOO matches GRPO; REINFORCE++ underperforms due to batch-level reward normalization inducing higher variance.
  • Reasoning traces become shorter and more focused over training (opposite pattern to math/code RL models).

Limitations

  • Planning performance modestly decreases (training objective targets single next keypoint, not long-horizon sequences).
  • Evaluation confined to table-top manipulation (5 training tasks from RLBench); generalization to contact-rich or dexterous tasks untested.
  • Environment metadata \(M\) (reference points, axis orientations, end-effector scale) must be manually specified per scene, limiting out-of-the-box applicability.
  • LLM-as-judge Pearson correlation for planning is only 0.33, making the planning evaluation metric unreliable.
  • Only one base model (7B) evaluated; no scaling experiments or comparison to larger open-source VLMs.
  • Training data is small (~7.5K pairs); how performance scales with more demonstrations is not studied.

Relevance to Vision-Language Models

Robot-R1 is a direct application of the DeepSeek-R1 RL-for-reasoning paradigm to the embodied domain, demonstrating that GRPO can elicit transferable spatial and movement reasoning in a 7B LVLM without the catastrophic forgetting that plagues SFT. The MCQA discretization trick is a broadly applicable design pattern for applying RL to tasks with continuous output spaces that resist direct reward shaping. The finding that embodied reasoning produces shorter, more focused chains—unlike the longer chains seen in math—challenges the assumption that more reasoning tokens universally improve performance and has implications for how VLMs should be trained for perception-action loops. The benchmark (Robot-R1 Bench) fills a gap between general VQA benchmarks and simulation-based robot success-rate metrics by directly testing the intermediate reasoning capabilities that connect vision-language understanding to low-level control.