Translation as a Bridging Action: Transferring Manipulation Skills from Humans to Robots¶
🕒 Published (v1): 2026-06-26 14:34 UTC · Source: Arxiv · link
Ask a follow-up
Open an assistant pre-loaded with this paper's context.
💬 Ask ChatGPT✦ Ask Claude
TL;DR¶
Proposes relative wrist translation (within head-camera frame) as a shared "bridging action" representation between humans and robots, avoiding noisy 6DoF human wrist poses that are misaligned with parallel gripper contact patterns. A π0-like vision-language-action model with interleaved action tokens and attention masking is trained in three stages (human pre‑training → human–robot co‑training → few‑shot robot post‑training), achieving 0.70 overall progress on 15 bi‑manual tasks.
Problem¶
Human action data is cheap, abundant, and diverse, making it a promising resource for scaling robot learning. However, prior work treats humans as just another 6DoF embodiment: hand‑pose estimates are noisy, and the contact patterns of human fingers differ fundamentally from parallel grippers, making wrist rotations unreliable and semantically misaligned. Learning rotation‑inclusive 6DoF action signals from human data is therefore sub‑optimal.
Method¶
Bridging action representation. The key insight is that both humans and robots act on what they perceive. The relative wrist translation is computed in the head‑camera frame:
where \(\mathbf{t}(\cdot)\) extracts the translation component. For bi‑manual setups both arms are concatenated, yielding \(a^{\text{3D-wrist}}_t \in \mathbb{R}^{k\times 6}\). This representation is robust to noisy rotation estimates, physically meaningful under a shared observation perspective, and embodiment‑agnostic.
Action components. Three action types are defined: \(a^{\text{3D-wrist}}\) (shared bridging translation), \(a^{\text{6D-eef}}\) (relative 6DoF end‑effector pose, \(\mathbb{R}^{k\times 12}\) per arm pair), and \(a^{\text{gripper}}\) (binary open/close, \(\mathbb{R}^{k\times 2}\)). Different data sources supervise different subsets (human data only provides \(a^{\text{3D-wrist}}\); robot data provides all three).
VLA with interleaved action tokens. A π0‑like vision‑language‑action model processes observations (head + two wrist cameras) and language instructions via a pre‑trained VLM, whose KV‑cache conditions an Action Transformer that generates action chunks via flow matching (\(\mathcal{L}_{\text{FM}} = \| \hat{v}(a^\tau_t, o_t, l, \tau) - (\epsilon - a_t) \|_2^2\)). Action tokens are ordered as \(a^{\text{3D-wrist}} \rightarrow a^{\text{6D-eef}} \rightarrow a^{\text{gripper}}\); missing components are masked in attention layers. Randomly substituting \(a^{\text{3D-wrist}}\) for \(a^{\text{6D-eef}}\) on robot data binds the bridging representation to executable actions.
Three‑stage training. 1. Stage I — Pre‑train on \(\sim\)600 h of human data (EgoDex + out‑sourced + in‑lab) with only \(\mathcal{L}^{\text{3D-wrist}}_{\text{FM}}\), 400 k iterations, batch size 1024. 2. Stage II — Co‑train with \(\sim\)72 h of generalized robot pick‑and‑place data + \(\sim\)3 h/task in‑lab human actions across 15 tasks, activating all three loss terms, 120 k iterations, batch size 256. 3. Stage III — Few‑shot post‑training with 10 robot trajectories per task, 25 k iterations.
Key Contributions¶
- A bridging action representation based on wrist translations (no rotations) that is shared between humans and robots, robust to noisy pose estimates, and embodiment‑agnostic.
- An interleaved action token design with attention masking that handles heterogeneous action components across data sources, enabling joint training on human and robot data.
- A three‑stage training strategy that exploits large‑scale human action pre‑training (\(\sim\)600 h) followed by human–robot co‑training and few‑shot robot post‑training.
- Real‑world validation on 15 bi‑manual manipulation tasks on the ByteMini robot platform, demonstrating consistent improvements from human‑only pre‑training and scaling with human data volume.
Results¶
- Overall progress/success on 15 tasks across four categories (progress shown; success in parentheses):
- Robot pick‑and‑place only (w/o human): 0.18 (0.12)
- Co‑train Stage II: 0.49 (0.31)
- Human pre‑train + co‑train Stage I+II: 0.60 (0.48)
-
Few‑shot post‑train Stage I+II+III: 0.70 (0.55)
-
Per‑category progress after full pipeline: Microwave 0.65, Drawer 0.72, Mug/Cup 0.70, Other 0.85.
- Random substitution of \(a^{\text{3D-wrist}}\) for \(a^{\text{6D-eef}}\) on robot data during co‑training is essential for transferring bridging knowledge to executable actions (Sec. 5.5).
- The method benefits from large‑scale pre‑training — the gap between Stage II and Stage I+II shows clear scaling with 600 h of human data.
Limitations¶
- Requires task‑specific in‑lab human data (\(\sim\)3 h per task) where operators mimic gripper postures, adding annotation overhead.
- Evaluated only on a single robot platform (ByteMini bi‑manipulator with parallel grippers); generalizability to other embodiments (e.g., dexterous hands, single‑arm systems) is unexplored.
- Discards wrist rotation entirely; some tasks that require precise orienting (e.g., screwing, pouring) may not transfer well through translation‑only actions.
- The 15‑task benchmark, while diverse, remains a limited set; scaling to hundreds of tasks and open‑vocabulary instructions is future work.
Relevance to Vision-Language Models¶
This work extends VLM representations into physical action generation via a π0‑style vision‑language‑action architecture, where pre‑trained vision‑language priors condition a flow‑matching action head. It demonstrates that large‑scale VLM backbones can be adapted for robotic control while jointly trained on heterogeneous data sources (human video + robot teleoperation). For researchers tracking VLMs, this paper illustrates a concrete pathway from language‑conditioned visual understanding to embodied manipulation, bridging the simulation‑to‑real gap through human data.