Skip to content

RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RoboRefer is a 3D-aware VLM for spatial referring—predicting a precise 2D point satisfying complex spatial constraints from RGB-D observations and natural-language instructions. It combines a disentangled depth encoder (SFT stage) with GRPO-based reinforcement fine-tuning using metric-sensitive process rewards (RFT stage) to enable multi-step spatial reasoning. On the authors' new RefSpatial-Bench, it surpasses Gemini-2.5-Pro by 17.4% absolute.

Problem

Existing VLMs handle single-step spatial understanding poorly in 3D (needing costly multi-view reconstruction or sharing RGB encoders with depth, causing modality interference) and lack multi-step spatial reasoning capability entirely—due to absent datasets and SFT's tendency to memorize rather than generalize compositional spatial constraints. No benchmark existed for evaluating multi-step spatial referring with up to 5 chained reasoning steps.

Method

Architecture: Separate RGB and depth encoders (each with its own projector) feed into a shared LLM (NVILA-2B/8B base). The depth encoder is initialized from the RGB encoder but updated independently, preventing modality interference while enriching 3D perception.

Two-stage training: 1. SFT — (a) Depth alignment: train only the depth projector on RGB-D annotations. (b) Full fine-tuning on RefSpatial (RGB + RGB-D jointly) plus instruction-tuning and referring datasets, teaching single-step spatial understanding and providing a cold-start for reasoning. 2. RFT — GRPO applied to the SFT checkpoint using multi-step reasoning data. Four reward signals are combined: Outcome Format Reward (\(R_{OF}\)), Point L1 Reward (\(R_P\), binary at ground-truth proximity), Process Format Reward (\(R_{PF}\), enforces step-format), and Accuracy Reward (\(R_{Acc}\), per-step metric error against key-step annotations with \(\alpha=0.25\)). Rewards are group-normalized into relative advantages; KL regularization keeps updates near the reference policy.

RefSpatial dataset: 2.5M samples / 20M QA pairs from three sources—OpenImages (2D, 466k filtered), CA-1M (3D embodied video, 100k filtered frames), and Infinigen+Objaverse simulation (3k verified assets). Covers 31 spatial relations (vs. 15 prior), hierarchical captions, top-down occupancy maps, and ground-truth step-by-step reasoning annotations (up to 5 steps) in simulation.

RefSpatial-Bench: 200 real-world images (100 Location + 100 Placement); 70%+ require multi-step reasoning; 77% involve unseen spatial-relation combinations; ground-truth precise masks used for evaluation.

Key Contributions

  • RoboRefer: first 3D-aware reasoning VLM for multi-step spatial referring, trained with sequential SFT→RFT.
  • Dedicated (disentangled) depth encoder design that avoids modality interference without requiring large RGB co-training data.
  • Metric-sensitive process reward functions for spatial RFT, enabling step-level intermediate precision refinement.
  • RefSpatial: large-scale dataset (20M QA, 31 spatial relations, explicit multi-step reasoning annotations).
  • RefSpatial-Bench: first benchmark for multi-step spatial referring evaluation (up to 5-step reasoning, unseen compositions).

Results

  • Single-step spatial understanding (SFT model, 89.6% average):
  • RoboRefer-8B-SFT (RGB-D): CV-Bench 96.90%, BLINKval 98.33% — surpasses Gemini-2.5-Pro (93.54% / 91.00%) and all spatial specialist models.
  • RoboRefer-2B-SFT outperforms NVILA-2B by 21.7% absolute.
  • Multi-step referring (RefSpatial-Bench, RFT model):
  • 2B-RFT: Location 52%, Placement 54%, Unseen 41.56% vs. Gemini-2.5-Pro: 46.96% / 24.21% / 27.14% — overall +17.4% average.
  • 2B-RFT exceeds 2B-SFT by 9.1% on unseen compositions, demonstrating RFT generalization.
  • Simulation (Open6DOR V2): RoboRefer 79.2% vs. SoFar 72.4%; execution time −27.5% vs. GPT-4o-based pipeline.
  • Real-world manipulation (Tab. 6): 80% / 60% success on two long-horizon cluttered tasks vs. 0% for OpenVLA and RoboPoint.
  • 2D referring (RefCOCO/+/g): Point-output RoboRefer-8B-SFT reaches 96.6/97.7/94.7 on RefCOCO val/testA/testB, outperforming Qwen2.5-VL-72B point-based evaluation.

Limitations

  • RFT applied only to the 2B model due to compute budget; 8B-RFT not evaluated.
  • Predicts 2D image-plane coordinates; 3D localization requires a separate depth map (DepthAnything V2) for back-projection.
  • Relies primarily on qualitative spatial relations; no direct prediction of quantitative 3D geometry or metric distances.
  • Human instructions are often ambiguous/brief; model struggles with intent inference when the correct target is non-obvious.
  • RefSpatial-Bench is small (200 images); may not capture the full diversity of real-world spatial configurations.

Relevance to Vision-Language Models

RoboRefer directly advances VLM spatial reasoning by showing that a dedicated depth encoder (vs. shared RGB-depth encoding) and RFT with step-level process rewards are the two critical ingredients for accurate, generalizable 3D spatial understanding—findings applicable beyond robotics to any VLM task requiring precise geometric grounding. The sequential SFT→RFT paradigm, where SFT provides structured spatial priors and RFT generalizes them via exploration, is a concrete recipe for instilling multi-step compositional reasoning into VLMs. RefSpatial's scale (20M QA, 31 relations, explicit chain-of-thought annotations) and RefSpatial-Bench's unseen-composition evaluation set a new standard for spatial reasoning benchmarking in the VLM community.