Skip to content

Human-Centric Transferable Tactile Pre-Training for Dexterous Robotic Manipulation

🕒 Published (v1): 2026-07-01 15:26 UTC · Source: Arxiv · link

Why this paper was selected

Tactile pre-training transferable to dexterous manipulation; cross-modal grounding for robotics

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TTP (Transferable Tactile Pre-Training) pre-trains a Vision-Language-Action (VLA) model on H-Tac, a new 160-hour, 135k-episode egocentric human tactile-action dataset, then post-trains on robot tasks using a shared tactile and action space to preserve human-to-robot transfer. A dual-expert architecture predicts future actions and future tactile signals jointly via flow matching, explicitly modeling contact dynamics. Experiments show improved fine-grained manipulation on both simulation benchmarks and real robots.

Problem

Existing VLA models are vision-centric and fail on contact-rich tasks because they lack tactile grounding; tactile datasets are too small and embodiment-specific; and prior tactile–VLA methods rely entirely on dynamics-agnostic post-training, leaving a large performance ceiling. No scalable human-to-robot tactile pre-training paradigm existed.

Method

Dataset (H-Tac). Three sub-datasets totaling 160 hours / 135k episodes: 1. HOI-Tac: ~106 hours, 11.5M frames from 12 public hand-object interaction datasets; per-vertex contact labels on 778-vertex MANO mesh projected to 351-taxel UniTacHand UV space. 2. DeskTask-Tac: 37.2 hours of real bimanual desktop tasks using RealSense cameras + tactile/MoCap gloves (RawHand and AprilTag pipelines); Spatial Tactile representation combines MANO pose \((\theta_h, \beta_h, R_h, t_h)\) with 351-D tactile vectors. 3. InternData-Tac: 17.8 hours of robot data augmented with a tactile recorder; contact patches projected onto the MANO surface via a Gaussian kernel to yield 351-D pseudo-contact signals for cross-embodiment compatibility.

Architecture. Built on top of BeingH-0.5, a VLA initialized from InternVL-3.5. Three experts share a transformer backbone via cross-attention: (1) understanding expert (vision + language), (2) action expert, (3) tactile expert. All experts share the same flow-matching training objective. The policy \(\hat{A}_t, \hat{O}^+_t \sim \pi_\theta(A_t, O^+_t \mid l, I_t, s_t, O^{hist}_t)\) predicts a \(K\)-step action chunk \(A_t \in \mathbb{R}^{K \times D_{act}}\) and future tactile readings \(O^+_t \in \mathbb{R}^{K \times D_{tac}}\).

Unified spaces. A 200-D action space covers EEF pose, dexterous hand joints, grippers, arm joints, mobile base, and MANO parameters, enabling one representation for humans and multiple robot embodiments. Tactile is standardized to \(D_{tac}=351\) MANO-UV taxels for both human and robot hands.

Flow matching loss. $\(\mathcal{L} = \lambda_{act}\mathcal{L}_{act} + \lambda_{tac}\mathcal{L}_{tac}\)$ where each \(\mathcal{L}_m = \mathbb{E}\|\hat{v}^m_\theta(x^m_\tau, \tau^m, H_{t,\tau^m}) - (x^m_1 - x^m_0)\|^2_2\).

Tactile-Action Manifold-Preserving Gating (MPG). A reliability gate \(g = \exp(-D/\tau_g)\), where \(D\) is the average sliced Wasserstein distance between the flow-conditioned context \(H_{t,\tau}\) and noise-free action/tactile anchors, modulates the residual enhancement of \(H_{t,\tau}\) before velocity decoding. This suppresses corrections when the context is out-of-distribution.

Key Contributions

  • H-Tac dataset: 160-hour, 300+ task, 135k-episode egocentric human tactile-action dataset (open-sourced).
  • First VLA tactile pre-training on human data: enables tactile-grounded priors before any robot-specific training.
  • Unified tactile and action space: bridges human hands, dexterous robot hands, and grippers within a single 351-D MANO-UV tactile space and 200-D action space.
  • Dual-expert flow matching: joint prediction of future actions and future tactile signals explicitly models contact dynamics.
  • MPG module: sliced Wasserstein-based gate stabilizes flow predictions under context distribution shifts.

Results

(All numbers from Table 2 and Figure 1; simulation benchmarks use "tactile proxy" because they lack real tactile sensors.)

LIBERO (simulation, 50 episodes/task): - TTP: 98.1% avg; TTP w/o pre-training: 97.4%; BeingH-0.5: 98.0%; π0.5: 96.8%.

LIBERO-plus (zero-shot generalization): - TTP: 75.7% avg; TTP w/o pre-training: 73.4%; BeingH-0.5: 76.0%; π0.5: 72.7%.

RoboCasa (50 trials/task, 24 tasks): - TTP: 55.1% avg; TTP w/o pre-training: 52.3%; BeingH-0.5: 53.9%; π0.5: 41.4%.

Real-world overall performance (Figure 1): - TTP: 32.72; TTP-scratch: 29.18; BeingH-0.5: 25.87; π0.5: 51.5 (shown for reference).

Limitations

  • Simulation benchmarks have no real tactile sensors; a "tactile proxy" (difference between last action and current proprioceptive state) substitutes, which may not faithfully reflect real tactile dynamics.
  • The paper text is truncated before full ablations and real-robot task breakdowns are reported.
  • H-Tac is dominated by grasping tasks (language statistics show "grasp" far outnumbers other verbs), potentially biasing learned priors.
  • Hardware heterogeneity (different piezoresistive sensors, visuo-tactile sensors) requires careful projection to the MANO UV space; sensor fidelity and calibration errors are not analyzed.
  • Data collection for DeskTask-Tac requires either tactile gloves or both tactile and MoCap gloves plus AprilTags, limiting scalability compared to purely vision-based data collection.

Relevance to Vision-Language Models

This work directly extends VLA models — which are VLMs augmented with action heads — by adding a third expert modality (tactile) at pre-training time rather than post-training, demonstrating that VLM-style scalable pre-training generalizes beyond vision and language to contact-rich physical signals. It validates the hypothesis that VLM architectures can implicitly align tactile signals with vision and language tokens when trained at scale on heterogeneous human data, a key open question for multimodal foundation models. The unified MANO-UV tactile space mirrors how VLMs use shared token spaces to bridge modalities, and the MPG gate provides a principled mechanism for handling out-of-distribution context that is applicable to any multimodal flow-matching model. Researchers tracking VLMs will find this a concrete demonstration of extending the VLA paradigm to a fourth sensing modality with human-data pre-training.