Skip to content

Robostral Navigate

🕒 Published (v1): 2026-07-22 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Robostral Navigate is an 8B VLM for embodied robot navigation that operates on a single monocular RGB stream, predicting waypoints via image-space pointing rather than metric coordinates. It introduces a prefix-caching training recipe that reduces token complexity from \(O(T^2)\) to \(O(T)\) and combines supervised fine-tuning with online RL (CISPO) to achieve new state-of-the-art results on R2R-CE (77.4% SR) and RxR-CE (75.1% SR), outperforming even depth-assisted systems.

Problem

State-of-the-art navigation systems require depth sensors, LiDAR, multi-camera rigs, or pre-built maps, limiting hardware compatibility, increasing deployment cost, and demanding environment-specific calibration. No prior single-camera VLM-based approach could match systems with privileged sensing, and training long-horizon navigation policies from simulation remained computationally prohibitive due to quadratic token scaling across episode time steps.

Method

Architecture: An 8B VLM initialized from a spatial-grounding model (pointing, localization) that takes a natural-language instruction and a history of monocular RGB frames, then outputs waypoints either as image-space pixel coordinates \((u, v, \Delta\theta)\) (pointing mode, when destination is visible) or metric displacements \((\Delta x, \Delta y, \Delta\theta)\) (fallback mode). A 121M-parameter diffusion transformer converts VLM waypoints into 10 Hz action chunks, which a motion-tracking controller converts to 100 Hz motor commands.

Prefix-caching training: Instead of treating each time step \(t\) as an independent sample (requiring re-encoding the full history \(I|O_0|\dots|O_t\) per step, \(O(T^2)\) tokens total), entire episodes are packed as single sequences \(I|O_0|a_0|\dots|O_n|a_n\) with \(O(T)\) token cost. A tree-based attention mask prevents the model from attending to ground-truth actions at previous steps during training (which would be unavailable at inference), yielding provably identical training signal at 22Ă— fewer tokens.

Online RL: After SFT, CISPO (group-relative advantage estimation) fine-tunes on a curated hard subset of 35k trajectories where the SFT policy fails. Reward is \(-\max(2, \text{dist\_to\_goal})\), clipped to prevent reward hacking near the goal and to incentivize the STOP action.

Data: 2.4 million simulated trajectories across 350k scenes (offices, residential, outdoor); robot morphology randomization (height 0.4–1.8m, camera pitch 0–25°) for cross-embodiment generalization.

Key Contributions

  • Image-space waypoint prediction via pointing, making the policy intrinsically robust to camera intrinsics and scene scale changes without recalibration.
  • Prefix-caching with tree-attention masking: reduces training token count by 22Ă— (\(O(T^2) \to O(T)\)) while preserving all learning signals, cutting training time from months to days.
  • Online RL with CISPO on a hard-task curriculum, adding ~4% SR improvement over SFT.
  • Cross-embodiment deployment using the same VLM and diffusion policy weights on two morphologically distinct robots (Galaxea R1, Hiwonder JetAuto).
  • State-of-the-art on both R2R-CE and RxR-CE using only monocular RGB, surpassing all depth/multi-camera baselines.

Results

All numbers are on validation-unseen splits.

R2R-CE: - Robostral Navigate: SR 77.4%, SPL 74.2%, NE 3.20m, OS 81.3% - Best prior single-camera (Qwen-RobotNav-4B): SR 66.9% → +10.5 pp - Best prior depth/multi-camera (Qwen-RobotNav-8B): SR 72.1% → +5.3 pp - SFT baseline → post-RL: 73.4% → 77.4% SR (+4.0 pp) on unseen

RxR-CE: - Robostral Navigate: SR 75.1%, SPL 68.7%, NE 3.47m - Best prior single-camera (Qwen-RobotNav-8B): SR 73.4%, SPL 63.5% → +1.7 pp SR, +5.2 pp SPL - Best prior depth-assisted baseline (Qwen-RobotNav-8B with depth): SR 76.5%, SPL 65.7% — Robostral exceeds on SPL (+3.0 pp) and NE (3.47m vs. 3.58m), nearly matches on SR - SFT baseline → post-RL: 71.2% → 75.1% SR (+3.9 pp)

Limitations

  • Evaluated only on indoor/simulated-to-real benchmarks (R2R-CE, RxR-CE); no outdoor or unstructured real-world benchmark results reported.
  • Metric displacement fallback mode still relies on robot-frame coordinates, partially reintroducing embodiment dependence in the ~10% of steps where the destination is outside the field of view.
  • The diffusion policy and motion-tracking controller require embodiment-specific tuning (robot height/radius inputs, platform controller); only the VLM weights are fully shared across platforms.
  • RL training requires substantial GPU orchestration (simulator + vLLM inference + distributed training concurrently); the systems engineering complexity is acknowledged but not fully characterized.
  • No ablation on real-world sim-to-real transfer gap; deployment on only two robot platforms is demonstrated.
  • RxR-CE evaluation is English-only despite RxR being a multilingual benchmark.

Relevance to Vision-Language Models

Robostral Navigate demonstrates that a general-purpose spatial-grounding VLM can be adapted into a state-of-the-art embodied navigator by reframing action prediction as a visual grounding task (pointing), directly leveraging capabilities—object localization, spatial reasoning—already present in the base VLM. The prefix-caching training technique is broadly applicable to any sequential VLM fine-tuning scenario involving long episode histories, offering a practical recipe for scaling VLM training on temporal data. The CISPO online RL stage provides a concrete example of post-SFT policy improvement for VLMs in interactive environments without reward model training. For VLM researchers, the work illustrates how operating in image space rather than task-specific coordinate systems enables a single model to generalize across sensor configurations—a principle relevant to any embodied or agentic VLM deployment.