X-VLA: Soft-Prompted Transformer as Scalable Cross-Embodiment Vision-Language-Action Model¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; X-VLA: soft-prompted transformer for cross-embodiment VLA

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

X-VLA is a cross-embodiment Vision-Language-Action model that uses lightweight learnable "soft prompt" embeddings—one set per data source—to absorb the heterogeneity (camera setups, action spaces, task distributions) across diverse robotic platforms. A 0.9B instantiation built on Florence-2-Large with a flow-matching Transformer backbone achieves state-of-the-art performance on six simulation benchmarks and three real-world robot platforms while requiring only 0.04% domain-specific parameters.

Problem¶

Cross-embodiment VLA training is undermined by severe distributional shifts arising from differences in hardware kinematics, camera configurations, action spaces, and task distributions across datasets. Prior methods address heterogeneity only at the action output stage (separate projection heads), leaving upstream visual and proprioceptive misalignment unresolved. HPT-style mid-stream projections destabilize pretrained VLM representations; natural-language hardware descriptions require manual scripting and don't scale.

Method¶

X-VLA assigns a set of $N=32$ learnable token embeddings $p_i \in \mathbb{R}^k$ to each dataset $D_i$ (7 sets during pretraining), collectively forming a Soft Prompt Library $P^H = \{p_i\}_{i=1}^H$. At runtime, prompts are selected by dataset ID and prepended to the concatenated multimodal token sequence before a stack of standard self-attention Transformer encoder blocks. The latent mapping $\Phi: \mathcal{H} \to \mathbb{R}^k$ from hardware config to prompt space is not predefined—it is randomly initialized and learned end-to-end alongside the backbone under the flow-matching objective: $$\mathcal{L}^{\text{FM}}_{\text{BC}}(\theta) = \mathbb{E}_{t, (o,A)\sim\mathcal{D}}\left[\|v_\theta(A_t, o, t) - (A - A_0)\|^2\right]$$ The encoding pipeline separates high-dimensional streams (main camera + language through a frozen Florence-2-Large VLM; auxiliary wrist views through a shared ViT) from a low-dimensional proprioceptive–action stream (joint positions + noisy action chunks concatenated and linearly projected). Action targets are standardized to EEF pose (xyz + Rotate6D rotation + binary gripper) with 30 anchor points over a 4-second horizon via temporal downsampling. Domain adaptation uses a two-step procedure: (1) prompt warm-up with backbone frozen, then (2) joint finetuning of backbone + new prompt, with reduced LR for the vision-language modules and soft prompts to prevent catastrophic drift.

Key Contributions¶

Soft Prompt mechanism for cross-embodiment heterogeneity: per-source learnable embeddings injected at the token level, adding only 0.04% domain-specific parameters.
X-VLA architecture: flow-matching policy using stacked standard Transformer encoders (not DiT) with disentangled multimodal encoding pipeline.
Two-step adaptation protocol: prompt warm-up then joint finetuning enables efficient PEFT (1% of parameters, 9M) for new embodiments.
Training recipe: aligned EEF action representation, intention abstraction via temporal downsampling to 30 anchors, and cross-domain balanced data shuffling.
Empirical ablation path (Table 1) isolating the contribution of each component to both pretraining validation error and downstream success rate.

Results¶

LIBERO: 93% success rate with 1% parameter finetuning (9M parameters), comparable to $\pi_0$ (3B parameters, 300× more).
Simpler-WidowX: 54% success rate with 1% parameter finetuning; ablation baseline (no pretraining) achieves 4.1% success rate; full system reaches 95.8% after two-step adaptation.
Pretraining validation error: reduced from 0.11 (baseline with custom LR) to 0.032 by adding all components including soft prompts and two-step adaptation.
Six simulation benchmarks: LIBERO, Simpler-WidowX, CALVIN, VLABench, RoboTwin2, and one autonomous driving benchmark—SOTA on most.
Three real-world robot platforms: AgiBOT, Franka, Agilex.
Cloth folding: 1,200 demos, average fold in under 2 minutes, comparable to closed-source larger models.
Scaling: validation error scales favorably ($R^2 = -0.925$) with increased compute/model size.
Soft prompts outperform HPT-style projection and language prompt baselines on training stability (Fig. 4).

Limitations¶

Soft prompts are indexed by dataset ID, requiring knowledge of source domain at inference; true zero-shot embodiment generalization (to an unseen robot with no dataset ID) is not demonstrated.
Evaluation of Phase II adaptation still requires at least a small set of target-domain demonstrations; few-shot or zero-shot transfer numbers are not provided.
The 7-platform pretraining dataset (290K episodes) is curated and filtered—scale and diversity relative to RT-2 or OpenVLA style web-scale pretraining is limited.
Architecture relies on Florence-2-Large as a fixed VLM backbone; the impact of VLM choice is not ablated.
Paper text is truncated before full scaling results (Fig. 6) are reported.

Relevance to Vision-Language Models¶

X-VLA directly extends VLMs into the action domain by using Florence-2-Large as the vision-language encoder and adapting prompt-tuning techniques—well-established in NLP and VLM fine-tuning—to handle embodiment heterogeneity. The soft prompt mechanism is a concrete instantiation of parameter-efficient adaptation (analogous to prefix tuning or prompt tuning for VLMs) applied to the multimodal grounding problem in robotics. For researchers tracking VLMs, the key insight is that per-source soft prompts placed in the token sequence before a Transformer encoder are more effective than either textual hardware descriptions (which stress the VLM's language head) or mid-stream projection layers (which corrupt frozen VLM representations)—a practically important design lesson for any VLM fine-tuning scenario with heterogeneous data sources.