RCT: A Robot-Collected Touch-Vision-Language Dataset for Tactile Generalization¶

🕒 Published (v1): 2026-06-30 14:05 UTC · Source: Arxiv · link

Why this paper was selected

Calandra; robot-collected tactile-vision-language dataset; enables tactile VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

RCT is a robot-collected touch–vision–language dataset of 29,279 tactile frames from full robot presses on 122 industrial materials using three DIGIT sensors, organized around full contact sequences rather than independent frames. It demonstrates that frame-random evaluation splits dramatically overestimate tactile generalization (17.7 pp inflation from contact-sequence overlap alone), and that held-out-material retrieval averages only 25.1 ± 6.1% — exposing novel-material generalization as a central open challenge.

Problem¶

Prior touch–vision–language datasets (TVL, Touch100k) evaluate tactile representations using frame-random train/test splits, which can place near-duplicate frames from the same robot press (same material, same sensor, same contact position, differing by only 0.10 mm indentation depth) in both training and test. This conflates contact-sequence memorization with genuine material-level generalization. The field lacked benchmarks that isolate whether a tactile representation transfers to unseen materials and unseen physical interactions — the condition robots face during open-world manipulation.

Method¶

RCT is collected by a robot arm with a rotating adapter holding three DIGIT vision-based tactile sensors, pressing each material at multiple discrete contact positions. Each press is recorded at fixed 0.10 mm depth steps, yielding 15–17 frames per contact sequence with per-frame contact force. The dataset contains 122 industrial reference materials across 7 categories (Plastic/Rubber, Paper/Cardboard, Metal, Textiles/Leather, Wood/Bamboo/Cork, Crafts, Small Items). Each material is paired with a studio photograph and human-annotated tactile descriptors from a fixed vocabulary (two annotators; Jaccard agreement 0.294). The evaluation protocol isolates five held-out dimensions: material, category, contact sequence, contact position, and sensor. The training recipe follows TVL: ViT-Base tactile encoder, OpenCLIP ViT-L/14 vision/text encoders, InfoNCE losses across modality pairs. A key methodological finding is that uniformly sampling 5 frames across the press depth (uniform5) outperforms using all ~16 frames for contrastive training.

Key Contributions¶

RCT dataset: 29,279 tactile frames from 122 materials × 3 DIGIT sensors × multiple contact positions, with full contact-sequence metadata and material-level vision/language annotations
Contact-sequence-aware evaluation protocols: five held-out settings (material, category, sequence, position, sensor) that isolate genuine generalization from frame-level overlap
Quantification of evaluation inflation: contact-sequence overlap inflates tactile→text Recall@1 by 17.7 pp; material overlap adds another 42.0 pp
TVL/HCT audit: 100% of HCT test contact sequences appear in training; a training-free raw-pixel nearest-neighbor baseline recovers the correct sequence in 98.3% of cases
Training insight: uniform sampling across a press improves contrastive training (68.18% vs. 62.44% full-density R@1) while using only 1/3 the data

Results¶

TVL reproduction: 82.21% tactile→vision R@1 on combined SSVTP+HCT (close to reported 81.7%)
Contact-sequence overlap effect: tactile→text R@1 drops from 80.01% (frame-random) to 62.33% (held-out-sequence) — a 17.7 pp drop from removing sequence overlap alone
Material holdout: further drop to 20.35% R@1 when test materials are also unseen (42.0 pp additional)
Held-out-material generalization: averaged 25.1 ± 6.1% R@1 over three held-out draws of K=20 materials
TVL/HCT split audit: all 279 HCT test contact sequences appear in training; median nearest same-sequence training frame is 1 depth-step away
Uniform5 sampling: 68.18% tac→text R@1 vs. 62.44% full-density vs. 59.52% deep5 on held-out-sequence evaluation
Material separability: RCT encoder margin = 0.297 vs. TVL encoder 0.048 (higher is better); category linear probe accuracy 49.8% vs. 34.2% TVL
Hard/soft prediction: all methods near majority baseline (~52.2%), including force features (52.6%)

Limitations¶

Limited scope: 122 industrial reference materials, 3 DIGIT sensors, flat-surface robot-press protocol only — does not cover arbitrary object geometries, curved surfaces, or dynamic exploratory motions (sliding, rubbing, grasping)
Material-level supervision: each material has one reference image and material-level descriptors, not per-frame visual targets or frame-specific tactile captions; strict one-to-one tactile→vision retrieval is not appropriate
Hardness remains difficult: binary hard/soft prediction stays near the majority baseline for all learned representations and even force features; suggests hardness perception requires richer interaction signals or different probing tasks
Moderate human annotation agreement: Jaccard 0.294 between two human annotators on tactile descriptors; VLM–human agreement even lower (~0.11), indicating visual appearance is only a partial proxy for tactile properties
Contact position imbalance: positions 1–4 densely sampled across all materials, later positions sparser

Relevance to Vision-Language Models¶

RCT directly extends the touch–vision–language (TVL) alignment paradigm by showing that evaluation protocol design — not just model architecture or data scale — determines whether reported scores reflect genuine generalization. The finding that contrastive learning benefits from subsampling correlated frames (uniform5) is relevant to any VLM training pipeline where temporal/spatial correlations create near-duplicate positives/negatives. For researchers tracking VLMs, RCT's core lesson applies broadly: held-out-distribution evaluation (materials, in this case) reveals capabilities that i.i.d. splits systematically overestimate.