BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Models¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

BridgeVLA is a 3D vision-language-action (VLA) model that resolves the distributional mismatch between 2D-pretrained VLMs and 3D robot manipulation by projecting point clouds into multi-view 2D orthographic images (input alignment) and predicting 2D heatmaps rather than token sequences for action localization (output alignment). A scalable heatmap pre-training stage on object grounding data bridges the gap between the VLM's token-prediction pre-training and the downstream spatial prediction requirement. The model achieves state-of-the-art on three simulation benchmarks and real-robot experiments with dramatically higher sample efficiency than prior VLA methods.

Problem¶

Existing 3D VLA models suffer from two compounding problems: (1) a distributional shift between the 2D image inputs used in VLM pre-training and the 3D inputs used at fine-tuning time, and (2) failure to exploit the spatial structure inherent in 3D data because actions are predicted as flat token sequences (next-token prediction) rather than in a spatially aligned space — the same design flaw that makes 3D structural priors ineffective and degrades sample efficiency. Pure 2D VLA models meanwhile require hundreds of trajectories per task.

Method¶

Backbone: PaliGemma (SigLIP vision encoder + Gemma transformer).

Phase 1 — 2D Heatmap Pre-training: Fine-tune the VLM on the 120K-image RoboPoint object detection split. Given an image and a text query, the model reconstructs the spatial grid from output image tokens, then applies a learned convex upsampling module (pixel-wise interpolation weights) to produce a full-resolution 2D heatmap trained with cross-entropy loss against Gaussian probability maps centered on ground-truth bounding boxes.

Phase 2 — 3D Action Fine-tuning: RGB-D observations are fused into a point cloud, then rendered as three orthographic projection images (top, front, right) — preserving the 2D image format the VLM was trained on. The VLM produces three 2D heatmaps; these are back-projected into 3D workspace point grids and the highest-scoring 3D point gives the translational action. Rotation (72-bin Euler discretization), gripper state, and collision flag are predicted by an MLP over global (max-pooled) and local (heatmap-peak) feature tokens from each view. A coarse-to-fine refinement runs a second forward pass on a zoomed-in cropped point cloud centered on the initial prediction. Loss: cross-entropy for translation and rotation, binary cross-entropy for gripper and collision.

No per-pixel 3D position features are injected into the VLM backbone (unlike SpatialVLA) — this is a deliberate design choice to avoid shifting the input feature distribution away from VLM pre-training.

Key Contributions¶

Input-output alignment paradigm: 3D point clouds projected to 2D orthographic images (input) and actions decoded as 2D spatial heatmaps (output), keeping both in the VLM's native 2D image space throughout pre-training and fine-tuning.
Scalable heatmap pre-training: Adapts a standard VLM to produce spatially structured outputs via object grounding on detection data, a step that proves essential for downstream language-guided generalization.
Ablation-validated design: Each component (heatmap output, no 3D position injection into backbone, heatmap pre-training) is rigorously ablated; removing any one causes large performance drops.
Exceptional sample efficiency: 95.4% real-robot success rate with only 3 demonstrations per task across 10+ tasks, where competing VLA methods (π0, SpatialVLA) fail.

Results¶

RLBench (18 tasks, 100 demos/task): BridgeVLA 88.2% avg success vs. RVT-2 81.4%, 3D Diffuser Actor 81.3%; average rank 2.03 vs. next-best 2.67. Best on 10/18 tasks; largest gains on Insert Peg (88.0% vs. 40.0%) and Sort Shape (60.8% vs. 35.0%).
COLOSSEUM (12-axis perturbation generalization): 64.0% vs. 56.7% prior SOTA.
GemBench (hierarchical generalization, 44 test tasks): Surpasses all baselines by average success rate.
Real robot, basic setting (10 demos/task, 13 tasks): BridgeVLA 96.9% vs. RVT-2 90%, SpatialVLA(50) 28.5%, π0 3.8%.
Real robot, 3 demos/task: 95.4% success rate (vs. π0 0%).
Real robot, 7 generalization settings averaged: BridgeVLA outperforms RVT-2 by 32%.
Ablation — heatmap vs. direct regression: removing heatmaps and using MSE position regression drops RLBench from 88.2% to 31.4%.
Ablation — adding 3D position to VLM input: drops RLBench from 88.2% to 56.2%.

Limitations¶

Place Cups task: weakest performance (still best among baselines) because target keypoints are frequently occluded in all three orthographic views; authors flag dynamic view selection as future work.
Unseen object category generalization: absolute success rate is low; robot often ignores novel target objects and moves directly to the destination, suggesting heatmap pre-training knowledge of novel objects does not transfer cleanly to manipulation keypoints.
Pre-training domain gap: pre-training images are predominantly third-person RGB captures, whereas fine-tuning inputs are synthetic orthographic point cloud renders — the gap limits category-level transfer.
Pre-training task scope: object localization alone does not cover manipulation keypoints (e.g., grasp points, destinations), limiting generalization to truly novel manipulation primitives.
Action decoder expressiveness: current MLP-based rotation/gripper prediction may be a bottleneck; authors identify diffusion-based decoding as a promising extension.

Relevance to Vision-Language Models¶

BridgeVLA directly addresses a core challenge in applying pre-trained VLMs to embodied tasks: how to preserve the rich representations learned during VLM pre-training when the downstream task requires spatial output formats (heatmaps) and non-standard inputs (3D point clouds). The finding that injecting 3D positional features into the VLM backbone hurts performance (−32 pp on RLBench) is a sharp result about representational alignment that generalizes beyond robotics. The two-stage protocol — adapt the VLM to a new output modality first (heatmap pre-training), then fine-tune end-to-end on the target domain — is a transferable recipe for VLM specialization. This work sits at the intersection of VLM adaptation, spatial grounding, and data-efficient policy learning, all active frontiers in VLM research.