Skip to content

See like a Robot: Robot-Centric Pointmaps for Vision-Language-Action Models

πŸ•’ Published (v1): 2026-07-13 00:00 UTC Β· Source: HuggingFace Β· link

Why this paper was selected

Robot-frame 3D pointmaps for VLA; fixes camera-frame mismatch, strong geometric inductive bias

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

πŸ’¬ Ask ChatGPT✦ Ask Claude

TL;DR

Vision-language-action (VLA) models observe scenes in the camera frame but predict actions in the robot frame, creating a coordinate mismatch that worsens under diverse camera viewpoints. This paper proposes robot-centric pointmaps β€” \(H \times W\) images whose pixels store 3D scene coordinates in the robot base frame β€” as a plug-in input that fuses with existing pretrained VLAs via element-wise token addition. On RoboCasa, pointmaps improve Ο€0.5 by +7.6 points and outperform dedicated camera-aware and 3D-augmented baselines.

Problem

Large-scale robot datasets aggregate demonstrations across institutions with heterogeneous camera setups, so a VLA must generalize the observation-to-action mapping across viewpoints rather than memorizing a single one. Existing solutions either (a) supply camera calibration cues but leave the frame conversion to the policy, (b) synthesize novel views (brittle for out-of-distribution cameras), or (c) build 3D scene representations that abandon the image grid expected by pretrained 2D VLAs. No prior method simultaneously provides metric robot-frame 3D geometry in image form.

Method

Pointmap construction. Given an RGB-D image from camera \(c\) with intrinsics \(K_c\), rotation \(R_c\), and translation \(t_c\) (camera-to-robot), each pixel \((u,v)\) is lifted to camera-frame 3D: $\(P_c^{cam}(u,v) = D_c(u,v)\,K_c^{-1}[u,v,1]^\top\)$ then transformed to the robot base frame: $\(P_c^R(u,v) = R_c\,P_c^{cam}(u,v) + t_c\)$ and re-centered on the current end-effector position \(t_{EE}\): $\(P_c^{EE}(u,v) = P_c^R(u,v) - t_{EE}\)$ This end-effector-centered pointmap expresses every scene point as a displacement from the gripper, making geometrically equivalent grasps appear identical regardless of their absolute workspace location.

Fusion. A separate pointmap encoder \(g_\phi\) (initialized from the VLA's RGB encoder \(f_\theta\)) maps \(P_c^{EE}\) to tokens of identical shape, which are added element-wise to the RGB tokens: $\(z_c = f_\theta(I_c) + g_\phi(P_c^{EE}) \in \mathbb{R}^{N_\text{tok} \times d}\)$ Element-wise addition (vs. concatenation) preserves spatial correspondence and introduces zero extra tokens.

Key Contributions

  • Robot-centric pointmaps as a pixel-aligned 3D input representation that retains the \(H \times W\) grid required by pretrained 2D VLAs.
  • Element-wise fusion of pointmap tokens onto RGB tokens β€” no new token sequence, no point-cloud encoder, no voxelization.
  • End-effector centering as the coordinate origin, shown to reduce viewpoint sensitivity by 6.7Γ— compared to robot-base centering (βˆ’0.3 vs. βˆ’2.0 points under viewpoint randomization).
  • Controlled ablation series isolating four design choices: pre-computing the robot-frame transform, image form vs. point cloud, coordinate origin, and robustness under increasing camera variation.

Results

  • RoboCasa (Ο€0.5): 55.3% β†’ 62.9% (+7.6 pp), outperforming the strongest camera-aware baseline KYC (59.1%) and strongest 3D-augmented baseline PointVLA (57.3%).
  • RoboCasa (SmolVLA): 37.2% β†’ 41.4% (+4.2 pp).
  • Pre-computation ablation: RGB + PlΓΌcker + Depth (same raw data, no pre-computed transform) achieves 31.6%; RGB + Pointmap achieves 34.7%, a +3.1 pp gap attributable solely to pre-computing the robot-frame transform.
  • Image form vs. point cloud: RGB + Pointmap (add) = 34.7%; RGB + PointCloud (PTv3, concat) = 32.8%; RGB + PointCloud (MLP, concat) = 24.2%.
  • Viewpoint robustness: Under high training-time camera variation, RGB drops 9.6 pp (34.5% β†’ 24.9%); RGB + Pointmap drops only 1.8 pp (37.6% β†’ 35.8%).
  • Real robot (Franka): +5.0 pp over RGB-only at a seen camera placement, +11.7 pp at a held-out unseen placement.

Limitations

  • Requires calibrated RGB-D input (accurate extrinsics \(R_c, t_c\) and per-pixel depth \(D_c\)) at both training and inference; calibration errors will degrade the robot-frame geometry.
  • Real-robot evaluation covers only four tasks (pick-and-place, stack blocks, open/close drawer) with 15 episodes each β€” limited diversity.
  • The approach does not apply to monocular RGB-only setups where depth is unavailable.
  • Baselines (KYC, OC-VLA, GeoVLA, PointVLA) were re-implemented by the authors due to unavailable official code, introducing potential implementation bias.
  • The real-robot Table 5 is truncated in the provided text, so full numerical comparisons against DP3 and RGB-only across all conditions cannot be verified from the excerpt.

Relevance to Vision-Language Models

This work directly addresses how pretrained VLMs (used as the backbone in VLAs) can absorb 3D spatial information without architectural surgery β€” a recurring challenge as VLMs are repurposed for embodied tasks. The key insight β€” that pixel-aligned geometric representations can be fused into the existing visual token stream through element-wise addition, reusing pretrained encoder weights β€” is transferable to any VLM architecture that ingests image tokens. The paper also provides a concrete diagnosis of why camera-frame observations limit VLM-based policies under distribution shift, which is relevant to anyone deploying VLMs in settings where the observation-action coordinate system is misaligned.