Wake up for Touch! Mask-isolated Tactile Alignment Learning in MLLMs¶

🕒 Published (v1): 2026-07-01 01:02 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

Tactile modality in MLLMs; mask-isolated alignment; ECCV 2026; fills physical-grounding gap

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Splash is a mask-guided tactile alignment framework for small multimodal LLMs (≤3B) that avoids catastrophic forgetting by confining all tactile adaptation to a dormant (low-importance) parameter subspace while keeping critical vision-language weights frozen. It achieves state-of-the-art visuo-tactile-language performance with a 3B model that surpasses prior 7B-scale methods, while fully preserving general VL reasoning. Training is single-stage with zero added LLM inference overhead.

Problem¶

Incorporating a new tactile modality into compact sMLLMs (≤3B parameters) degrades pretrained vision-language capabilities—catastrophic forgetting. The parameter budget is too small to absorb a new sensory modality without overwriting existing VL weight configurations. Prior VTL methods either use large 7B backbones unsuitable for edge robots, or employ two-stage PEFT (e.g., LoRA) that still perturbs the full parameter manifold.

Method¶

Splash partitions LLM parameters into a frozen critical subspace and a trainable dormant subspace using a VL-relative importance metric. Given a calibration set of 128 VL samples, each weight \(W_{i,j}\) in the LLM's linear layers receives an importance score:

\[S_{i,j} = |W_{i,j}| \cdot \|x_i\|_2\]

where \(|W_{i,j}|\) is weight magnitude and \(\|x_i\|_2\) is the \(\ell_2\)-norm of the corresponding input activation. A binary mask \(\mathbf{M}_{i,j} = \mathbf{1}(S_{i,j} < \tau_j(s))\) identifies the bottom-\(s\)\% as dormant; first and last transformer blocks are always frozen. Gradient updates are restricted via Hadamard product: \(\theta^* \leftarrow \theta - \eta(\mathbf{M} \odot \nabla_\theta \mathcal{L})\).

The tactile front-end is a ViT-Tiny (5.8M params, ImageNet-pretrained) with a two-layer MLP projector, trained fully alongside the dormant subspace in a single unified stage (no separate alignment pretraining). Natural image, tactile contact image, and text tokens are concatenated into one autoregressive sequence. Default sparsity ratio \(s = 60\%\).

Key Contributions¶

Mask-isolated parameter partitioning: frozen critical subspace anchors VL knowledge; dormant subspace absorbs tactile signals without additive modules—zero LLM inference overhead.
Single-stage unified training: eliminates the two-stage align-then-finetune pipeline used by TVL and UniTouch, reducing training cost.
3B model outperforms 7B baselines: Splash-3B exceeds TVL-LLaMA2-7B on all three VTL benchmarks.
First comprehensive VL-preservation analysis for VTL alignment: evaluates post-adaptation retention on MMMU, MathVista, MME, and MMBench.

Results¶

VTL benchmarks (LLM-as-judge, 1–10 scale, avg over SSVTP / TVL / TacQuad): - Splash-1B (InternVL2.5-1B): 5.01 avg - Splash-3B (Qwen2.5-VL-3B): 4.91 avg - TVL-Qwen2.5-VL-3B (LoRA): 4.50 avg - TVL-LLaMA2-7B: 4.31 avg - UniTouch-LLaMA1-7B: 3.74 avg - Splash-3B on TacQuad (distribution-shift): 4.86 vs. (IA)³ 3.84 and TVL-LoRA 4.22

VL preservation (Splash-3B vs. TVL-Qwen LoRA vs. zero-shot baseline): - MMMUval: 55.3 vs. 50.0 vs. 53.1 (zero-shot) - MathVista: 65.3 vs. 52.8 vs. 62.3 (zero-shot) - MMBench-EN: 78.0 vs. 76.8 vs. 79.1 (zero-shot) - MMBench-CN: 76.9 vs. 76.5 vs. 78.1 (zero-shot) - UniTouch collapses on VL tasks (MMMUval 26.6, MathVista 9.0)

Sparsity ablation: \(s=60\%\) is the Pareto-optimal point; \(s=100\%\) (full fine-tuning) collapses VL (MMMUval 22.0, MathVista 38.4).

Limitations¶

Evaluated only on DIGIT-sensor tactile data; generalization to other sensor types (GelSight, Tacto) not demonstrated beyond TacQuad's distribution-shift subset.
Splash-1B achieves the best average VTL score (5.01) yet shows lower VL retention than Splash-3B (e.g., MMBench-EN 70.2 vs. 78.0), suggesting the method scales better with more capacity.
The calibration set for importance scoring (128 CC12M samples) is fixed; sensitivity to calibration set choice is not ablated.
Paper text is truncated; full ablations (e.g., mask computation cost, training time vs. LoRA) may be incomplete in the provided excerpt.

Relevance to Vision-Language Models¶

Splash directly addresses a core tension in multimodal VLM extension: how to add a new sensory modality to a pretrained vision-language model without degrading existing capabilities, a challenge that parallels continual learning and modality expansion in VLMs broadly. The importance-scoring approach (magnitude × activation norm, inherited from Wanda-style pruning) offers a lightweight, architecture-agnostic mechanism applicable to any transformer-based VLM backbone. For VLM researchers, the result that \(s=60\%\) dormant-subspace updates can match or exceed LoRA on both new-modality acquisition and VL retention challenges the assumption that additive PEFT methods are safest for preserving pretrained representations. This work contributes a concrete framework for non-destructive sensory grounding of VLMs—relevant to embodied AI, robot manipulation, and any domain requiring physical property inference that vision alone cannot resolve.