ForceVLA: Enhancing VLA Models with a Force-aware MoE for Contact-rich Manipulation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ForceVLA augments Vision-Language-Action (VLA) models with 6-axis force-torque sensing as a first-class modality via a novel Mixture-of-Experts fusion module (FVLMoE), enabling contact-aware robotic manipulation. Built on top of π0, it dynamically routes multimodal tokens (vision, language, force) through sparse expert subnetworks during action decoding. It achieves 60.5% average task success across five contact-rich tasks, a 23.2% improvement over the π0 baseline.

Problem

Existing VLA models rely exclusively on visual and linguistic inputs and fail at contact-rich manipulation tasks (insertion, assembly, tool use) that require fine-grained force modulation—especially under visual occlusion or dynamic uncertainty. Naive force concatenation into existing architectures provides only marginal gains and can degrade performance in compact architectures like π0-fast, because raw force tokens lack corresponding large-scale pretraining alignment.

Method

ForceVLA extends the π0 framework (PaliGemma VLM backbone + flow-based action decoding) with FVLMoE, a post-VLM Mixture-of-Experts fusion module:

1. Input encoding: Visual and language inputs are processed by a frozen SigLIP/PaliGemma VLM to produce embeddings \(E_{VL} \in \mathbb{R}^{N_{VL} \times D}\). The 6D force-torque reading \(f_{raw} \in \mathbb{R}^6\) is linearly projected to a single force token \(E_F \in \mathbb{R}^D\).
2. FVLMoE fusion: Concatenated sequence \(E_{in} = [E_{VL}; E_F]\) passes through a shared self-attention encoder layer, then a sparse MoE layer with \(E=4\) expert MLPs and top-\(k=1\) routing per token. A learned gating network dispatches each token to the most relevant expert subnetwork.
3. Action injection: The final \(H_{action}\) fused tokens from FVLMoE are element-wise added to the proprioceptive/noisy-action suffix tokens in the flow matching denoiser, directly modulating trajectory generation at each denoising step.

The key architectural decision—introducing force after the VLM, not before or concurrently—preserves pretrained visual-linguistic feature distributions while enabling specialized force-context routing.

Key Contributions

• FVLMoE module: a post-VLM sparse MoE fusion layer that dynamically integrates 6D force-torque tokens with VL embeddings via learned routing, enabling phase-aware contact adaptation during action generation.
• ForceVLA-Data: 244 real-world trajectories (~140k synchronized timesteps) spanning 5 contact-rich tasks with synchronized RGB-D vision, proprioception, and 6-axis force-torque signals; released in LeRobot format with teleoperation and conversion tools.
• End-to-end VLA policy treating force as a first-class modality, demonstrating 23.2% average improvement over π0-base and up to 80% success on plug insertion.
• Ablation showing post-VLM late fusion via MoE is strictly superior to early fusion (MoE before VLM: 0% success) and naive late concatenation (60% vs 80%).

Results

• Average success rate: ForceVLA 60.5% vs. π0-base w/o F 37.3% (+23.2%), π0-base w/ F 40.2%, π0-fast w/o F 31.0%, π0-fast w/ F 14.2%.
• Per-task highlights: Insert USB 83%, Insert Plug 80%, Pump Bottle 93%, Wipe Board-1 67%/Board-2 64%, Peel Cucumber 25% (all ForceVLA).
• Cucumber peeling: ForceVLA 14.12 cm avg peel length (vs. 13.17 and 10.27 for baselines), 7 strokes to clean (vs. 10 and 14).
• Generalization (Table 2): ForceVLA 63.78% average across 5 held-out variation conditions vs. best baseline π0-fast w/o F at 52.78%; 90% under visual occlusion.
• Ablation (Table 3, plug insertion): baseline 45%, linear before VLM 55%, MoE before VLM 0%, concatenate after VLM 60%, ForceVLA 80%.

Limitations

• Uses estimated external wrench from joint torques rather than a direct high-fidelity force-torque sensor measurement, potentially losing precision in extreme haptic sensitivity scenarios.
• Requires integrated (typically high-cost) force-torque sensing hardware, limiting accessibility; portability to low-cost retrofitted sensors is under investigation but not yet validated.
• No simulation training or evaluation; the real-world-first approach avoids sim-to-real gap but restricts dataset scale and task diversity.
• Dataset is small (244 trajectories, ~50 demonstrations per task) and limited to 5 tasks on a single robot platform.

Relevance to Vision-Language Models

ForceVLA directly extends the VLA paradigm—where VLMs serve as the perception backbone for robotic policies—by formalizing force-torque sensing as a peer modality alongside vision and language rather than an auxiliary input. The FVLMoE architecture demonstrates a principled strategy for grafting new sensing modalities onto frozen pretrained VLMs without disrupting their learned representations, a pattern directly applicable to other modality extensions (tactile, audio, depth). For researchers tracking VLMs, this work surfaces the brittleness of visual-only grounding in physical interaction settings and proposes MoE-based late fusion as a scalable solution for multimodal embodied agents.