Skip to content

3D HAMSTER: Bridging Planning and Control in Hierarchical Vision Language Action Models through 3D Trajectory Guidance

๐Ÿ•’ Published (v1): 2026-06-30 00:00 UTC ยท Source: HuggingFace ยท link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

๐Ÿ’ฌ Ask ChatGPTโœฆ Ask Claude

TL;DR

Hierarchical VLA models mismatch their planners (2D pixel space) against 3D-native low-level policies (point clouds), causing geometrically distorted "graffiti" trajectories when 2D waypoints are naively lifted to 3D. 3D HAMSTER closes this gap by augmenting a VLM with a dedicated depth encoder and a dense depth reconstruction loss to predict metric 3D trajectories \((u, v, d)\) directly. The resulting guidance is fed to a pointcloud-based policy in a shared metric space, improving robustness under distribution shift in simulation and on a real Franka Panda arm.

Problem

Prior hierarchical VLA frameworks (e.g., HAMSTER) pair a 2D-outputting VLM planner with 3D-native, pointcloud-based low-level policies. Bridging the two requires assigning depth to each 2D waypoint by sampling the scene surface at that pixel, yielding a "graffiti effect": the trajectory clings to visible geometry rather than tracing the intended free-space path through 3D space. This representational misalignment degrades manipulation performance, especially under appearance-altering perturbations (lighting, texture) and novel spatial/language conditions.

Method

3D Trajectory Planner. Built on Qwen3-VL-8B-Instruct, the planner is augmented with a separately initialized depth encoder (from LingBot-Depth). RGB tokens and depth tokens are each projected into LLM embedding space, fused element-wise, and passed to the transformer backbone, which autoregressively generates waypoints as \((u_t, v_t, d_t)\) tuples in JSON format.

Depth Reconstruction Loss. Depth tokens \(z_D\) are routed through a lightweight decoder \(f_{\text{dec}}\) to reconstruct the full depth map \(\hat{D} = f_{\text{dec}}(z_D)\), supervised by \(\mathcal{L}_{\text{depth}} = \|D - \hat{D}\|_1\). The combined loss is: $\(\mathcal{L} = \mathcal{L}_{\text{LM}} + \lambda \mathcal{L}_{\text{depth}}, \quad \lambda = 0.1\)$ This dense scene-level regularizer prevents depth features from degrading under sparse trajectory supervision.

Two-Stage Training. Stage 1 freezes all encoders and the LLM; only the depth projector and depth decoder are trained to align depth representations with the pretrained VLM space. Stage 2 freezes both encoders and fine-tunes the rest via LoRA (rank 64) on a curated 8-source data mixture (~6.15M samples) spanning real/sim robot demonstrations, spatial QA, and VLM preservation data.

Trajectory-conditioned 3D Low-Level Policy (3DFA). Predicted waypoints \(\tau\) are unprojected to world coordinates using known camera intrinsics/extrinsics: \(p_{\text{cam}} = d \cdot K^{-1}[u,v,1]^\top\), \(p_{\text{world}} = R\,p_{\text{cam}} + t_{\text{cam}}\). These 3D waypoints are appended to the scene point cloud with color-coded temporal indices and learnable modality embeddings \((e_{\text{traj}}, e_{\text{scene}})\). The policy predicts action chunks via rectified flow matching over this unified representation.

Key Contributions

  • A hierarchical VLA framework that eliminates 2D-to-3D unprojection artifacts by having the VLM planner directly output metrically reliable 3D end-effector trajectories consumable by a pointcloud policy.
  • A depth encoder + dense \(\ell_1\) depth reconstruction loss that regularizes VLM hidden states to preserve metric scene geometry beyond what sparse waypoint supervision alone provides.
  • DroidSpatial-Bench: a 3D trajectory prediction benchmark derived from 148 held-out DROID pick-and-place episodes evaluated at \(\delta \in \{5, 10\}\) cm tolerances.
  • Trajectory-scene fusion via temporal color coding, learnable modality embeddings, and priority-preserving point cloud subsampling to guarantee trajectory guidance is never dropped.

Results

  • DroidSpatial-Bench (3D trajectory prediction, Both metric):
  • Full 3D HAMSTER: 41.9% @ 5 cm / 65.5% @ 10 cm vs. RoboBrain-2.5-8B (39.2% / 60.1%), Gemini-3.0-Pro (16.2% / 29.7%), GPT-5.2 (2.7% / 16.2%), Sonnet-4.6 (0.7% / 2.0%)
  • Ablation: base Qwen3-VL-8B (RGB only) yields 0.7% / 0.7%; adding 3D data โ†’ 27.7% / 50.0%; adding depth encoder โ†’ 42.6% / 62.8%; adding \(\mathcal{L}_{\text{depth}}\) โ†’ 41.9% / 65.5%
  • Colosseum simulation (11 tasks, 14 perturbation axes, success rate %):
  • 3DFA (no guidance): 53.8% (none), 39.0% (MO), avg lower across all variations
  • 3DFA + HAMSTER (2D): 49.5% (none), 38.1% (MO)
  • 3DFA + 3D HAMSTER: 62.9% (none), 49.4% (MO), 36.3% (RO) โ€” largest gains under appearance-altering perturbations (lighting: 54.4% vs. 38.8% / 38.0%)
  • Real-world Franka Panda: 3D HAMSTER outperforms 2D-guided HAMSTER and ฯ€0.5 across all four generalization axes (language, spatial, visual, combined) on button pressing, pouring, and pick-and-place tasks (specific numbers not fully included in provided text).

Limitations

  • Requires calibrated RGB-D cameras with known intrinsics/extrinsics; metric depth from a monocular RGB-only setting is explicitly insufficient (near-zero baseline accuracy).
  • Depth maps for two training sources (InternData-M1, RefSpatial) are synthetically generated by MoGe-2, introducing potential domain gap between predicted and real sensor depth.
  • The planner is evaluated with a single VLM backbone (Qwen3-VL-8B); generalization across backbone scales or architectures is not demonstrated.
  • Concurrent works (RoboTracer, RoboBrain 2.5) address the same 3D-aware planning direction but were not fully comparable (RoboTracer withheld; RoboBrain tested as trajectory predictor only, not in closed-loop manipulation).
  • Simulation training uses only 100 demonstrations per task, which may limit absolute performance ceilings.

Relevance to Vision-Language Models

This paper directly addresses a critical failure mode of applying VLMs to embodied settings: VLMs are pretrained on rich geometric data yet are constrained to 2D outputs by hierarchical robot frameworks, creating a metric mismatch with 3D-native controllers. The depth encoder augmentation and dense reconstruction loss are lightweight but effective techniques for injecting metric geometric grounding into a frozen VLM backbone without eroding general vision-language capabilities โ€” a design pattern broadly applicable to any VLM deployment requiring spatial precision. The result that proprietary frontier VLMs (Gemini-3.0-Pro, GPT-5.2) achieve only 16โ€“30% on a 3D trajectory benchmark despite strong general performance quantifies the gap between general VLM spatial reasoning and metric 3D spatial reasoning for robotic planning. For researchers tracking VLMs, this benchmarks how well current SoTA models generalize to metric 3D grounding tasks and proposes a transferable recipe for bridging that gap.