villa-X: Enhancing Latent Action Modeling in Vision-Language-Action Models

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

villa-X: enhanced latent action modeling for VLA models; architectural advance

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VILLA-X is a Vision-Language-Latent-Action (ViLLA) framework that improves robot manipulation policies by physically grounding latent actions via proprioceptive supervision and integrating them into VLA pre-training through a joint diffusion architecture with two specialized action experts. It achieves state-of-the-art performance on SIMPLER simulation benchmarks and two real-world robotic setups, with zero-shot generalization to unseen embodiments and open-vocabulary concepts.

Problem

Existing latent action models compress motion semantics from visual signals alone, leaving latent representations physically ungrounded — subtle but control-critical motions (e.g., end-effector rotations, gripper actuation) produce small pixel changes and are underweighted. Separately, prior integration strategies (LAPA-style weight initialization, Go-1-style autoregressive discrete tokens) fail to transfer latent action knowledge effectively into robot action prediction at inference time.

Method

VILLA-X has two components trained in three sequential stages (LAM pretraining → ACT pretraining → embodiment finetuning):

Latent Action Model (LAM): Extends the standard IDM/FDM architecture with an auxiliary proprioceptive Forward Dynamics Model (proprio-FDM). Given current state \(q_t\) and latent token \(z_t\), it predicts future proprioceptive states and actions \(K\) steps ahead: $\((\hat{q}_{t+1}, \ldots, \hat{q}_{t+K},\, \hat{a}_{t+1}, \ldots, \hat{a}_{t+K}) = \text{proprio-FDM}(q_t, z_t, c_e)\)$ An embodiment context vector \(c_e = f(\text{dataset ID}, \text{control frequency})\) — learnable dataset embeddings concatenated with sinusoidal frequency encodings — prevents embodiment-specific dynamics from leaking into the shared latent space. The LAM jointly optimizes image reconstruction, proprioceptive prediction, and VQ commitment losses; for human video without proprioception labels, the proprio term is omitted. Continuous VQ codebook centers are used as latent actions.

ACTor Module (ACT): A three-expert architecture (VLM encoder, ACT-latent, ACT-robot) with blockwise causal attention. The policy is factorized as: $\(\pi(a_{t:t+m-1}, z^K_{t:t+(n-1)K} \mid o_t, l, q_t, c_e) = \pi_\text{robot}(a \mid z^K, o_t, l, q_t, c_e)\cdot \pi_\text{latent}(z^K \mid o_t, l)\)$ Both action types are modeled jointly via conditional flow matching. A stochastic masking strategy (50% of training steps: all robot-to-latent attention masked; otherwise 50% of latent tokens randomly masked) prevents the robot branch from shortcutting around the latent representation.

Key Contributions

• Proprio-FDM auxiliary decoder that grounds latent actions in physical dynamics by jointly supervising visual and proprioceptive forecasting across heterogeneous embodiments via an explicit embodiment context vector.
• Joint diffusion policy (ACT) with ACT-latent and ACT-robot experts connected via blockwise causal attention, replacing initialization-only or discrete-token integration schemes with structured mid-level conditioning.
• Zero-shot latent planning capability: after scaled pre-training, ACT-latent generalizes to unseen robot morphologies and open-vocabulary symbolic icons without any finetuning.

Results

• SIMPLER — Google Robot (post-finetuning): VILLA-X avg. 77.7% vs. \(\pi_0\) 58.7%, OpenVLA-OFT 63.0%, GR00T-N1.5 57.9%, MoTo 59.2%; Pick task reaches 98.7%.
• SIMPLER — WidowX Robot: VILLA-X avg. 62.5% vs. LAPA 57.3%, GR00T-N1.5 62.0%, Magma 44.8%.
• LAM probing (LIBERO, L1 error): w/pp variant produces more low-error samples than wo/pp at every threshold; max-L1 analysis over 8 action dimensions.
• ACT integration ablation (SIMPLER): VILLA-X (58.5% Google avg., 40.8% WidowX avg.) outperforms LAPA-style (43.8% / 1.0%) and Go-1-style (43.9% / 36.5%); removing LAM entirely drops Google avg. to 35.0%.
• Real-world (Realman gripper, 5 tasks, 10 trials each): VILLA-X outperforms all baselines in both standard and color-generalization settings.
• Real-world (XArm + 12-DoF XHand, 4,000-trajectory dataset): Evaluated on dexterous manipulation; VILLA-X achieves top performance (full numbers appear in paper tables).

Limitations

• Proprio-FDM requires robot proprioceptive labels; human video data skips this supervision, leaving grounding asymmetric across data sources.
• Zero-shot embodiment generalization is demonstrated qualitatively via world-model rendering, not quantitative success-rate evaluation on unseen embodiments.
• Embodiment context \(c_e\) relies on dataset ID embeddings, which do not generalize to entirely new datasets without adding new learned embeddings.
• The framework is evaluated on manipulation-only tasks; locomotion or whole-body control scenarios are untested.
• Extension to other structural cues (end-effector keypoints, hand pose estimation) is described as future work, not validated.

Relevance to Vision-Language Models

VILLA-X directly extends VLMs into the VLA paradigm by using a pre-trained vision-language backbone as the VLM encoder and showing that its open-vocabulary semantic representations survive robot pre-training intact (verified by the open-vocabulary symbol experiment). The joint diffusion policy with blockwise causal attention is a concrete architectural recipe for tightly coupling VLM features with mid-level latent plans and low-level action prediction, which is directly relevant for researchers studying how VLMs can be grounded to continuous control. The physically grounded latent action space provides a reusable intermediate representation layer between visual-language perception and motor control, a design pattern increasingly important as the VLM community scales multimodal pre-training to embodied tasks.