FoMoVLA: Bridging Visual Foresight and Motion Guidance for Vision-Language-Action Models¶

🕒 Published (v1): 2026-07-16 09:04 UTC · Source: Arxiv · link

Why this paper was selected

Visual foresight + motion guidance in VLA models; explicit forward prediction addresses reactivity

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

FoMoVLA augments Vision-Language-Action (VLA) models with two complementary training-time auxiliary objectives — compact future feature prediction via EMA-supervised foresight tokens and sparse 2D point tracking via a frozen CoTracker teacher — coupled through a future-conditioned cross-attention (FCCA) module. At inference, all auxiliary branches are discarded with negligible overhead (+9.4 ms latency, +0.1 GB GPU memory). The framework achieves state-of-the-art on LIBERO (98.8% avg.), RoboCasa GR-1 Tabletop (56.9%), and strong zero-shot OOD generalization on LIBERO-Plus (80.5%).

Problem¶

Existing VLA models are purely reactive: they map instantaneous observations and language instructions to actions without modeling future scene dynamics. Prior visual foresight methods either predict dense pixel-level future frames (high overhead, captures irrelevant static content) or compact keyframe-level goal representations (misses the continuous motion path to the goal). Point tracking methods capture motion but lack a goal-state anchor. Neither line integrates foresight with motion guidance in a jointly coherent representation.

Method¶

FoMoVLA adds three training-only branches on top of a pretrained VLA backbone (StarVLA-GR00T):

1. Point Tracking Branch. Text tokens are placed before image tokens so each image-token hidden state is conditioned on the full language instruction. $N = 64$ sparse grid points (one per 8×8 patch cell) are tracked across the $T$-step action chunk by a frozen CoTracker-v3 teacher, generating ground-truth displacements $d^\star \in \mathbb{R}^{N \times T \times 2}$ and visibility labels $v^\star$. Two lightweight MLP heads predict $\hat{d}$ and $\hat{v}$ from image-token hidden states. The motion loss is: $$\mathcal{L}_\text{track} = \mathcal{L}_\text{disp} + \mathcal{L}_\text{vis} + \lambda_\text{smooth}\mathcal{L}_\text{smooth}$$ where the smoothness term penalizes second-order temporal differences $\Delta^2 \hat{d}_{n,t}$ on visible points.

2. Future Feature Prediction Branch. $K=16$ learnable <Foresight> tokens are appended to the VLM input. Their hidden states are supervised to reconstruct visual features of the final chunk frame $o_{t+T}$ produced by an EMA teacher (momentum $\mu=0.999$). A 2-layer ViT MAE decoder expands the $K$-token bottleneck with $M-K=48$ mask tokens and optimizes a patch-wise cosine similarity loss: $$\mathcal{L}_\text{foresight} = 1 - \frac{1}{M}\sum_{i=1}^{M}\frac{\hat{z}_i \cdot z_i^\star}{\|\hat{z}_i\|\|z_i^\star\|}$$

3. Future-Conditioned Cross-Attention (FCCA). A zero-initialized multi-head cross-attention module (8 heads) injects foresight context $H_\text{fut} \in \mathbb{R}^{K \times d}$ into the spatial image features $H_\text{vis} \in \mathbb{R}^{N \times d}$ before the displacement predictor: $$\tilde{H}_\text{vis} = H_\text{vis} + \text{MHA}(\text{LN}(H_\text{vis}),\,\text{LN}(H_\text{fut}),\,\text{LN}(H_\text{fut}))$$ Zero initialization preserves pretrained features at the start of training. The overall loss is $\mathcal{L} = \mathcal{L}_\text{action} + \lambda_1 \mathcal{L}_\text{foresight} + \lambda_2(\mathcal{L}_\text{disp} + \mathcal{L}_\text{vis} + \lambda_\text{smooth}\mathcal{L}_\text{smooth})$ with $\lambda_1=0.1$, $\lambda_2=0.3$, $\lambda_\text{smooth}=0.1$. The action head is a DiT trained via conditional flow matching.

Key Contributions¶

Joint training framework that combines future feature prediction (goal-state) and sparse 2D point tracking (motion path) as complementary spatio-temporal supervision for VLA models.
FCCA module that conditions motion prediction on predicted future features, enforcing global-local consistency between foresight and trajectory estimates.
Inference-time zero overhead for auxiliary branches; only the $K=16$ foresight tokens remain in the input sequence at test time.
Demonstrated scalability across multiple policy heads (flow-matching, $\pi$-head, OFT) without architecture-specific coupling.

Results¶

LIBERO (4 suites, 20 rollouts/task, T=8): FoMoVLA achieves 98.8% avg. (Spatial 98.4, Object 99.6, Goal 99.4, Long 97.6), the highest reported average across all baselines.
Best prior methods: Spatial Forcing 98.5%, Cosmos Policy 98.5%, LingBot-VA 98.5%, LangForce 98.4%.
Ablation: Base backbone alone reaches 96.5%; adding FCCA over both branches gives +2.3% over base.
RoboCasa GR-1 Tabletop (24 tasks, 50 rollouts/task, T=16): FoMoVLA achieves 56.9%, vs. StarVLA-GR00T backbone 47.8% (+9.1%) and next-best StarVLA-OFT 48.8%.
LIBERO-Plus (zero-shot OOD, 7 perturbation axes, no fine-tuning): FoMoVLA achieves 80.5%, matching Abot-M0 (which uses additional pretraining) and outperforming StarVLA by +6.4%.
Largest gains: language perturbation (+5.5% over StarVLA), background perturbation (+9.2% over StarVLA).
Scalability: FoMoVLA applied to StarVLA-π improves from 95.7% to 97.9% (+8.2% on LIBERO-Long); applied to StarVLA-OFT from 96.6% to 98.0%.
Ablation (LIBERO-Long): Without FCCA, avg. drops from 98.8% to 98.3%; shared goal tokens collapse to 94.8% avg. (85.8% on Long); sparse 8×8 grid outperforms dense 16×16 (98.8% vs. 98.1%).

Limitations¶

Generalization to novel camera viewpoints and initial robot poses is weak, as acknowledged by the authors: camera and robot-state OOD gains on LIBERO-Plus are modest, attributed to training on fixed viewpoints and 2D-only motion cues.
Point tracking supervision is 2D only; it cannot directly encode 3D spatial reasoning or depth-dependent motion.
Dependence on a frozen off-the-shelf point tracker (CoTracker-v3) for training-time supervision; quality of supervision is bounded by tracker accuracy.
FCCA adds a cross-attention module; although zero-initialized, it introduces additional parameters and a coupling dependency between the two auxiliary objectives.
Evaluated only on tabletop manipulation benchmarks; generalization to contact-rich, deformable-object, or mobile manipulation settings is not demonstrated.

Relevance to Vision-Language Models¶

FoMoVLA is directly relevant to the question of how VLMs can be enhanced for embodied control: it shows that the visual representation bottleneck in VLMs (not language or action decoding) is the primary limiting factor, and addresses it via structured spatio-temporal auxiliary supervision during fine-tuning. The FCCA design is a concrete mechanism for injecting predictive world-model knowledge into frozen or lightly tuned VLM backbones without modifying the action decoder, making the approach backbone-agnostic and applicable to the growing class of VLA architectures built atop vision-language pretrained models. For researchers tracking VLMs, the framework demonstrates that future-feature distillation via EMA teachers and sparse geometric supervision are complementary inductive biases that improve visual grounding without pixel-level generation overhead — an efficiency-focused alternative to world-model-as-video-generator approaches.