4D-VLA: Spatiotemporal Vision-Language-Action Pretraining with Cross-Scene Calibration¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

4D-VLA addresses the problem that existing VLA pretraining methods use incomplete inputs (single RGB frame), causing coordinate system chaos and state chaos that disperses the conditional action distribution. It augments a VLM backbone with depth-derived 3D coordinate embeddings and an adaptive memory bank sampling strategy for historical frames, yielding strong spatiotemporal grounding. On LIBERO it achieves 88.6% average success rate, +12.1% over OpenVLA.

Problem¶

Diverse robotic pretraining datasets contain samples recorded from heterogeneous viewpoints with the robot body frequently occluded (67% of DROID samples have the robot base out of frame). This produces coordinate system chaos—actions are defined in robot frame but the model has no spatial grounding—and state chaos—visually similar frames correspond to different actions because temporal context is absent. Both inflate the variance of the target action distribution At(·) and degrade pretraining efficiency.

Method¶

Backbone. InternVL-4B (vision encoder + MLP projector + LLM decoder) with an appended two-layer MLP action head predicting [Δx, Δθ, grip].

Spatial-aware vision tokens. Per-frame depth D is back-projected into world-frame 3D coordinates Pw using camera intrinsics K and extrinsics [R|T]. A learnable positional embedding ES encodes Pw and is element-wise added to the vision encoder output before the MLP projector, producing spatial vision tokens eST that carry metric 3D context.

Memory bank sampling (MBS). From a temporal window of n=20 historical RGB-D frames, MBS greedily selects k=5 maximally dissimilar frames by maintaining a similarity queue: a candidate frame replaces the most similar existing frame whenever it is less similar to its neighbor, ensuring diverse temporal coverage without dense uniform sampling.

Temporal positional encoding. Because MBS is non-uniform, each sampled frame gets a learnable relative-offset token eT prepended before its spatial vision token; these pairs [eT|eST] are concatenated with text tokens and fed to the LLM.

Loss. L = Lt + Lr + Lg + λd·Ld where Lt,Lr are L2 on translation/rotation, Lg is BCE on gripper, and Ld = ‖d(Δx̂)−d(Δx)‖² is a directional loss on the unit-normalized translation vector.

MV-Bench. A new multi-view simulation benchmark built on LIBERO-SPATIAL with 6 training and 6 testing viewpoints spanning 270°, evaluating In-View and Cross-View generalization.

Key Contributions¶

Identification and formalization of coordinate system chaos and state chaos as root causes of inefficient VLA pretraining.
Spatial vision token generation via back-projected 3D coordinate embeddings fused into existing vision encoder features with negligible architectural overhead.
Memory bank sampling: an adaptive, similarity-driven historical frame selector that outperforms uniform sampling with fewer frames (k=5, n=20).
MV-Bench: first multi-view manipulation benchmark for evaluating VLA spatial generalization and cross-viewpoint transfer.
Full validation on LIBERO simulation, real-world Franka tasks, and multi-view real-world experiments.

Results¶

LIBERO (simulation, success rate): 4D-VLA 88.6% avg vs. OpenVLA 76.5% (+12.1%), SpatialVLA 78.1%, Octo 75.1%; LIBERO-LONG 79.1% vs. OpenVLA 53.7% (+25.4%).
MV-Bench In-View: 4D-VLA 81.0% vs. OpenVLA 52.2% (+28.8%).
MV-Bench Cross-View (avg): 4D-VLA 73.8% vs. OpenVLA 50.5%.
Real-world (4 tasks, avg success rate): full 4D-VLA 85.63% vs. OpenVLA 27.70%; base VLA without pretraining 15.67%; +Pretraining alone 47.00%; +Coord. encoding 63.67%; +History (MBS) 65.88%.
Real-world multi-view: 4D-VLA avg 55% vs. OpenVLA 18% across in-view and cross-view settings.

Limitations¶

Requires RGB-D input (depth sensor), adding hardware constraints not present in RGB-only pipelines.
Memory bank sampling's performance is sensitive to the temporal window size n; the number of sampled frames k has comparatively minor effect, suggesting headroom for analysis of optimal n.
Pretraining conducted on a single dataset (DROID); generalization to broader cross-embodiment datasets is not evaluated.
The Base VLA without pretraining underperforms OpenVLA, indicating the backbone alone (InternVL-4B + MLP head) is not competitive without the full pretraining regime.

Relevance to Vision-Language Models¶

4D-VLA extends the VLM-to-VLA paradigm by showing that standard VLM backbones (InternVL-4B) can be upgraded to strong robot controllers by injecting metric 3D structure—depth-derived coordinate embeddings—into the visual token stream, without modifying the LLM weights during pretraining. This is directly relevant to the VLM community because it demonstrates a lightweight, geometry-aware input augmentation that meaningfully reduces action distribution variance across heterogeneous datasets, a bottleneck that purely semantic VLM features cannot resolve. The MV-Bench contribution also creates a rigorous test for spatial understanding that complements existing VLM spatial reasoning benchmarks. For researchers tracking VLMs, this work establishes that 4D spatiotemporal grounding—not just richer language or larger scale—is a critical axis for embodied VLM generalization.