Skip to content

Xiaomi-Robotics-1: Scaling Vision-Language-Action Models with over 100K Hours of Real-World Trajectories

🕒 Published (v1): 2026-07-16 16:02 UTC · Source: Arxiv · link

Why this paper was selected

Xiaomi robotics; 100K hrs real VLA trajectories; massive-scale grounded VLM-action

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Xiaomi-Robotics-1 is a VLA foundation model pre-trained on over 100K hours of real-world UMI-collected manipulation trajectories, using a scalable VLM-powered auto-labeling pipeline that converts raw video segments into state-transition language descriptions. A two-stage recipe (pre-training on UMI data → cross-embodiment post-training) yields strong out-of-the-box generalization and data-efficient fine-tuning, with clear scaling laws in both data volume and model size.

Problem

Prior robot VLA models are bottlenecked by teleoperation data, which is slow, hardware-bound, and limited in diversity. Existing labeling approaches require manual segmentation and annotation, making them prohibitive at scale. The field lacks a robot foundation model trained on truly massive, open-world, heterogeneous manipulation data.

Method

Architecture: Mixture-of-Transformers (MoT) coupling Qwen3-VL (VLM backbone, 2.1B–8.8B params) with a Diffusion Transformer (DiT, 470M–1.5B params). The VLM encodes observations and language; the DiT generates action chunks via flow-matching conditioned on the VLM's KV cache and robot proprioceptive state:

\[\mathcal{L}_\text{Flow}(\theta) = \|v_\theta(o_t, l, s_t, \tilde{a}^\tau_{t:t+H}, \tau) - u(\tilde{a}^\tau_{t:t+H}, a_{t:t+H}, \tau)\|_2^2\]

An auxiliary Choice Policy head on the VLM (winner-takes-all over \(K\) candidate chunks) provides regression supervision \(\mathcal{L}_\text{Regression}\) to accelerate convergence; crucially, these action-related tokens are excluded from the DiT's attention to prevent shortcut copying.

Pre-training: 100K+ hours of egocentric UMI gripper data auto-labeled with Qwen3.5-27B, which captions fixed-length segments as state-transition descriptions (e.g., "the sponge moves from the sink to the rack"). Co-trained with VL data at a 9:1 trajectory-to-VL ratio with \(\lambda=0.1\) weight on \(\mathcal{L}_\text{NTP}\).

Post-training: ~10K hours of cross-embodiment data (7.2K hours in-house mobile manipulator + dual-arm, 1K hours instruction-labeled UMI, open-source datasets Bridge V2/RT-1/DROID). Language conditioning shifts from state-transition descriptions to imperative human instructions. Action spaces unified via relative delta end-effector poses with a masked unified action vector for heterogeneous embodiments.

Key Contributions

  • Scalable auto-labeling pipeline annotating 100K+ hours of UMI trajectories in ~2 weeks using a VLM (Qwen3.5-27B) in a producer-consumer architecture
  • Two-stage pre-training/post-training recipe that transfers UMI-learned action representations to robot embodiments with imperative instruction conditioning
  • Mixture-of-Transformers architecture with a Choice Policy auxiliary head and explicit exclusion of action tokens from the DiT's attention
  • Empirical demonstration that pre-training scaling laws (data volume and model size) transfer monotonically to post-training out-of-the-box performance
  • State-of-the-art results on RoboCasa365 (57.6% vs. prior best 46.6%) and RoboDojo (20.07 vs. prior best 13.07)

Results

  • Post-training out-of-the-box (4 real-robot tasks, unseen environments):
  • Without action pre-training: 26% overall success rate
  • With 100% pre-training data (5B): 75% overall success rate
  • 10B variant: 79% overall; shoe tidying task reaches 92% (vs. 58% for 2B)
  • Downstream fine-tuning (4 novel dexterous tasks, <10 hours/task average):
  • Xiaomi-Robotics-1: 75% avg. success rate, 90% avg. progress
  • Ď€0.5 baseline: 40% avg. success rate, 66% avg. progress
  • Printer refilling specifically: 70% (ours) vs. 20% (best baseline) in low-data setting
  • Simulation benchmarks:
  • RoboCasa365: 57.6% success (prior SOTA 46.6%)
  • RoboDojo: 20.07 avg. score (prior SOTA 13.07)
  • Also SOTA on RoboCasa and VLABench (exact numbers not provided in the excerpt)
  • Scaling: Validation MSE decreases monotonically with data scale (2B→5B→10B) and model size; data volume is the primary bottleneck at current scales

Limitations

  • Scaling experiments use only up to 20K hours of the full 100K+ hour corpus due to compute budget constraints; full-data pre-training scaling behavior is inferred, not directly measured at the largest scale
  • Model size effect is less pronounced than data volume effect, suggesting current model capacity may already exceed dataset expressivity at the tested data scales
  • Post-training out-of-the-box evaluation covers only 4 tasks; generalization breadth is demonstrated qualitatively
  • Action space unification via masked unified vectors may lose information for heterogeneous embodiments with structurally different kinematics
  • Paper text is truncated; fine-tuning results for high-data setting and full simulation benchmark numbers are not fully visible

Relevance to Vision-Language Models

Xiaomi-Robotics-1 directly extends VLM capabilities into embodied action generation by using Qwen3-VL as a frozen-then-jointly-trained backbone within a MoT architecture, showing that VLM representations can be efficiently repurposed for low-level motor control via a coupled DiT. The auto-labeling pipeline—where a large VLM annotates manipulation videos with natural language—demonstrates a scalable data flywheel that could generalize to any vision-language grounding task requiring dense video captioning. The finding that VLM-scale pre-training scaling laws transfer to robot action quality directly motivates continued investment in larger VLM backbones for embodied AI. This work is relevant to researchers tracking VLMs as it exemplifies the emerging VLA paradigm where VLM pretraining is the core transferable representation, not a peripheral component.