Skip to content

Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs

🕒 Published (v1): 2026-07-02 00:00 UTC · Source: HuggingFace · link

Why this paper was selected

Task-agnostic VLA pretraining; novel approach to VLA data bottleneck via motion pre-training

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TAP (Task-Agnostic Pretraining) decouples "how to move" from "what to do" in Vision-Language-Action model training by first learning physical priors via a self-supervised Inverse Dynamics objective on cheap, unlabeled robot interaction data, then aligning those priors to language with minimal expert demonstrations. On the SIMPLER benchmark, TAP-20k matches models trained on 1M+ expert trajectories while using orders of magnitude less labeled data (+10% absolute over standard behavior cloning). On a real WidowX robot, TAP retains 25% success under severe camera perturbations where internet-scale baselines collapse to 0%.

Problem

VLA models are bottlenecked by costly expert demonstrations (observation–instruction–action triplets). Current pipelines conflate two distinct objectives—learning physical motor competence and learning semantic task grounding—forcing both to be solved from expensive, language-annotated expert data. Cheap, unlabeled robot interaction data (off-task trajectories, autonomous play) is discarded entirely, despite containing rich physical priors.

Method

TAP is a two-stage framework grounded in a Decomposition Hypothesis: action generation factorizes into (1) physical affordance acquisition ("how to move") and (2) semantic intent grounding ("what to do"), where only the latter requires language.

Stage 1 — Task-Agnostic Pretraining: A VLM backbone (Qwen2.5-VL 3B + SigLIP encoder) is trained with a self-supervised Inverse Dynamics (ID) objective on unlabeled trajectories from two sources: repurposed task-irrelevant dataset trajectories (Bridge/OXE with labels discarded) and autonomously collected robot "play" data. Given observation pair \((o_t, o_{t+1})\), the model predicts the intervening action: $\(\hat{a}_t \leftarrow f_\theta(\phi(o_t), \phi(o_{t+1}))\)$ $\(\mathcal{L}_\text{ID}(\theta) = \mathbb{E}_{(o_t, a_t, o_{t+1}) \sim \mathcal{D}_\text{TAP}} \left\| \hat{a}_t - a_t \right\|_2^2\)$ This forces the visual encoder to attend to dynamic elements (end-effector motion, object displacement) and ignore static background, building dynamics-aware representations without any language supervision.

Autonomous play data is collected via a constrained procedural pipeline: voxel-downsampled safe pose library → stochastic waypoint sampling → contact heuristic (forced descent if end-effector stays above \(z_\text{thresh}\)) → boundary-aware Gaussian noise injection.

Stage 2 — Task-Specific Alignment: The pretrained model is finetuned on a small expert set \(\mathcal{D}_\text{expert}\) via standard behavior cloning, switching conditioning from future-frame \(o_{t+1}\) to language instruction \(l\): $\(\hat{a}_t \leftarrow f_\theta(\phi(o_t), \psi(l))\)$ The backbone and action head are reused; Stage 2 learns only a lightweight semantic-to-dynamics projection.

Action space: delta-pose end-effector \(a_t \in \mathbb{R}^7\) (\(\Delta x, \Delta y, \Delta z\), axis-angle orientation, gripper scalar).

Key Contributions

  • Decomposition Hypothesis: formal argument that physical motor priors and semantic task grounding are separable learning objectives, only the latter needing language supervision.
  • TAP framework: two-stage pipeline that unlocks task-agnostic data (discarded trajectories + autonomous play) for VLA pretraining via inverse dynamics.
  • Autonomous play collection pipeline: procedural trajectory generation with contact heuristics and a safe pose library, enabling cost-free embodiment-specific data at scale.
  • Empirical validation that physical competence (partial/grasping success) learned in Stage 1 directly predicts full-task success gains in Stage 2, confirming the bottleneck structure.
  • Demonstration that TAP-induced representations are distribution-shift robust, outperforming internet-scale models under visual perturbations.

Results

  • SIMPLER benchmark (simulation, WidowX environment):
  • TAP-20k: Avg-All 33.32% vs. Standard BC 23.15% (+10.17% absolute)
  • TAP-20k Avg-Partial 45.82% — matches Octo (42.30%) and approaches \(\pi_0\) (53.10%), despite no language in Stage 1
  • TAP-20k exceeds OpenVLA (7.75%) and RT-1-X (3.03%) on Avg-All, both pretrained on ~1M expert trajectories
  • Monotonic scaling: 8k→14k→20k episodes = 24.47%→30.21%→33.32% Avg-All
  • Real-world WidowX (200 expert demos, 30h autonomous play):
  • Standard setup, push task: TAP 75% vs. BC 55% vs. NORA 85%
  • Background Texture Shift, push task: TAP 65% vs. NORA 55% vs. BC 0%
  • Viewpoint Variation, push task: TAP 25% vs. NORA 0% vs. BC 0%
  • Viewpoint Variation, carrot task: TAP 15% vs. NORA 0% vs. BC 0%
  • Average across all conditions, push task: TAP 61% vs. NORA 56% vs. BC 21%
  • (Abstract states: TAP retains 25% success under camera perturbations where internet-scale baselines collapse to 0%)

Limitations

  • Stage 1 data scale in simulation is modest (up to 20k episodes); authors note TAP scaling has not saturated, leaving performance gains on the table.
  • Real-world play data collection still requires an operator for initial workspace coverage and periodic object shuffling (~30 min intervals).
  • Evaluation confined to two real-world tasks on a single robot platform (WidowX 250s); generalization to other embodiments or dexterous manipulation is untested.
  • Stage 2 still requires some expert demonstrations (200 real-world, 5k simulation); the minimum viable expert set is not ablated.
  • The Decomposition Hypothesis assumes action-space alignment between play and expert tasks; domain gaps with very different task morphologies (e.g., bimanual, tool use) are not addressed.

Relevance to Vision-Language Models

TAP directly challenges the assumption that VLMs used as VLA backbones must be trained end-to-end on language-annotated data from the start, showing that the visual encoder of a VLM can first acquire physically grounded, dynamics-aware representations via inverse dynamics before any language conditioning is applied. This has implications for how visual representations in VLMs are evaluated: partial success (grasping) becomes a probe for physical grounding quality independent of semantic alignment. The work also connects to broader debates about modality-specific pretraining in VLMs—analogous to how text and vision encoders are often pretrained separately before multimodal alignment—and suggests that for embodied VLMs, a motor-prior pretraining stage could replace or supplement internet-scale language-supervised pretraining for physical tasks.