Skip to content

The Moving Eye: Enhancing VLA Spatial Generalization via Hybrid Dynamic Data Collection

🕒 Published (v1): 2026-07-02 15:30 UTC · Source: Arxiv · link

Why this paper was selected

Hybrid dynamic data collection for VLA spatial generalization; actionable recipe for robot VLM training

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLA models suffer from shortcut learning—exploiting fixed camera–robot–object correlations rather than true spatial understanding—causing brittle generalization to novel viewpoints. This paper proposes a dual-arm data collection system where one arm manipulates while the other serves as a continuously moving environmental camera, yielding a hybrid static/dynamic training distribution. A Moving:Multi-Fixed mixing ratio of 1:3 (the "Golden Ratio") achieves best spatial generalization across multiple architectures.

Problem

VLA models trained on fixed or discretely-varying camera poses overfit to three spurious couplings: Camera-Base (memorizing robot appearance relative to background), Camera-Object (viewpoint-specific object recognition), and Object-Position (fixed relative poses between target and receptacle). Existing solutions—multi-view diversity, 3D pretraining, novel-view synthesis—do not systematically break all three couplings. Simply adding more static viewpoints is insufficient: the model still memorizes discrete pose–action mappings without learning invariant spatial representations.

Method

A dual-arm robot setup is used: the manipulation arm executes tasks via a wrist camera; a second arm (Airbot) continuously moves an environmental camera within a bounded workspace \(W\). Three view configurations are defined—Fixed (single static pose), Multi-Fixed (static per episode, varies across episodes), and Moving (continuous trajectory at 0.05 m/s linear, mean 0.198 rad/s angular). Training data \(D_{\text{train}}\) is formed as:

\[D_{\text{train}} = \frac{k}{k+1}D_{\text{Multi-Fixed}} + \frac{1}{k+1}D_{\text{Moving}}\]

where \(k\) controls the Multi-Fixed fraction. Object configuration diversity (randomized target–receptacle relative pose) is explicitly injected to break Object-Position coupling independent of viewpoint strategy. The optimal ratio for Gr00t is Moving:Multi-Fixed = 1:3 (the "Golden Ratio"), found empirically. Auxiliary pen-task Moving-View data is also transferred to robustify static policies on unseen multi-object tasks, decoupling semantic learning (from fixed-view task data) from spatial perception learning (from moving-view auxiliary data).

Key Contributions

  • Real-world realization of a continuous moving-view data collection system on physical dual-arm hardware, extending prior simulation-only paradigms (e.g., MOVE).
  • Identification and isolation of three shortcut coupling types (Camera-Base, Camera-Object, Object-Position) with controlled diagnostic experiments confirming each.
  • Demonstration that a hybrid Moving+Multi-Fixed distribution at 1:3 outperforms all pure-view conditions for Gr00t, with pure Moving data actually underperforming pure Multi-Fixed (54.8% vs. 80.5%) due to convergence instability.
  • Cross-architecture universality: ACT, Diffusion Policy, Pi0, and Gr00t all benefit from the hybrid strategy, though optimal ratio differs per architecture.
  • Cross-task spatial transfer: Moving-View data collected on a low-cost auxiliary pen task robustifies static policies on entirely different object classes.

Results

  • Camera-Base coupling (Exp. 1): Fixed-view baseline collapses from 85.0% (ID) → 43.0% (OOD-Moving); Mixed Data: 86.0% (ID) → 83.0% (OOD-Moving).
  • Object-Position coupling (Exp. 2): Multi-Fixed baseline: 95.0% (ID) → 71.9% (OOD-shifted holder ±1 diameter); Mixed 1:3: 91.9% (ID) → 90.6% (OOD).
  • Composition sweep (Exp. 3, Gr00t, Moving Test): Pure Moving 54.8%, 1:1 mix 83.3%, 1:3 mix 89.0% (best), pure Multi-Fixed 80.5%.
  • Architecture universality (Exp. 4): Diffusion Policy +26.8% improvement with hybrid data; Pi0 +13.8%; all architectures tested (ACT, Diffusion, Pi0, Gr00t) show gains.
  • Cross-task transfer: Mixed data (50% Fixed multi-task + 50% auxiliary Moving pen data) consistently outperforms pure Fixed multi-task baseline on Moving Test across all sample sizes evaluated.

Limitations

  • Optimal Moving:Multi-Fixed ratio is architecture-specific and requires per-model empirical search; no principled formula is provided.
  • Camera-Base decoupling is achieved via viewpoint variation, not robot-base movement—true base-pose variation is not evaluated.
  • Experiments are confined to tabletop pick-and-place with a single robot platform (SO-101 + Airbot); generalization to dexterous manipulation or other embodiments is unverified.
  • Pure Moving-View data is convergence-unstable; the method is sensitive to the frame rate and camera speed regime—very fast or abrupt viewpoint motion is explicitly declared out of scope.
  • Object-Position diversity requires deliberate re-arrangement during collection, adding human operator overhead not quantified in the paper.

Relevance to Vision-Language Models

This work directly exposes a failure mode of VLAs (Pi0, Gr00t, ACT) that is architectural-agnostic: all models shortcut on visual regularities introduced by standard data collection pipelines. For researchers tracking VLMs/VLAs, the key insight is that data distribution design—not only model capacity or pretraining—governs spatial generalization, challenging the prevailing assumption that scale or multi-view pretraining resolves viewpoint fragility. The cross-task transfer result (spatial know-how transferring from an auxiliary task) has implications for how VLA pretraining datasets should be curated: diverse-viewpoint auxiliary data may substitute for large-scale task-specific collection. The shortcut taxonomy (three coupling types) also provides a diagnostic vocabulary useful for evaluating future embodied VLMs.