Scaffolding Dexterous Manipulation with Vision-Language Models¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper proposes using off-the-shelf VLMs to generate coarse 3D keypoint-and-wrist trajectory "scaffolds" for dexterous robot hands, replacing the need for human demonstrations or hand-crafted reward functions. A residual RL policy trained in simulation tracks these scaffolds, achieving 72–81% average success across 8 tasks and successful sim-to-real transfer at up to 90% success rate.

Problem¶

Training dexterous manipulation policies is bottlenecked by two complementary problems: imitation learning requires scarce, hard-to-collect demonstrations for multi-fingered hands, while RL requires carefully engineered, task-specific reward functions. Prior trajectory-tracking RL methods resolve reward design but reintroduce dependence on large demonstration datasets (often thousands of teleoperated examples), limiting scalability and adaptability to new scene configurations.

Method¶

Given a language instruction (e.g., "hammer once") and an RGB-D image, an off-the-shelf VLM (Gemini 2.5 Flash Thinking) performs three steps: (1) keypoint detection — identifies k task-relevant 2D keypoints in the image, lifted to 3D via depth; (2) trajectory generation — produces n=20 waypoints for each keypoint and the wrist pose in world coordinates; (3) interpolation — linearly interpolates to full-horizon trajectories. These VLM-generated "scaffolds" τ supervise a low-level residual RL policy (3-layer MLP, PPO) trained in simulation (ManiSkill + Allegro Hand), which predicts wrist offsets Δw and finger joint targets. The dense reward combines mean keypoint tracking error (exponential) and a contact-incentive term. Plans are randomized across N=100 initial conditions per task; few-shot prompting with successful plans iteratively refines VLM outputs at inference.

Key Contributions¶

Framework eliminating demonstrations and hand-crafted rewards for dexterous manipulation by substituting VLM-generated keypoint scaffolds.
Hierarchical decomposition: VLM as high-level planner (coarse 3D trajectories) + residual RL as low-level controller (finger control + wrist offsets).
Task-agnostic dense reward derived purely from keypoint tracking error, applicable across all tasks.
Iterative few-shot refinement loop: successful plans fed back as in-context examples to improve VLM planning.
Sim-to-real transfer on a 16-DoF Allegro hand + KUKA arm without any real-world demonstrations or reward engineering.

Results¶

Simulation (8 tasks, ManiSkill, 2000 eval episodes/task, 3 seeds):
Zero-shot: 63% mean success rate
Few-shot (3 in-context examples): 72% mean success rate
Iterative refinement (3 iterations): 81% mean success rate
Oracle (scripted plans): ~99% on most tasks; method approaches oracle on several tasks
IKER (VLM-coded reward baseline): substantially lower on most tasks (e.g., 13% Apple, 23% Bottle vs. 45%/63% ours zero-shot)
Real-world (20 rollouts each):
Place Bottle onto Plate: 90%
Slide Box to Bottle: 85%
Hammer Three Times: 65%
Dominant failure mode: incomplete trajectory tracking (26% of rollouts); keypoint detection errors: 5%.
Performance saturates at ~10 waypoints for most tasks; hammer requires 20.

Limitations¶

Assumes rigid-body object interactions; cannot handle deformable objects without non-rigid keypoint tracking.
Orientation inference from keypoints is unreliable for small objects with few non-collinear keypoints.
High-level VLM planner is open-loop with respect to the low-level controller — no feedback from RL into planning.
VLM reasoning latency of 1–2 minutes precludes real-time replanning.
Generalization to entirely new task categories (beyond the 8 trained tasks) remains a challenge.
Sim-to-real gap persists for high-precision tasks; relies on domain randomization and state-based low-level observations.

Relevance to Vision-Language Models¶

This paper demonstrates a concrete, working role for VLMs as geometric and semantic planners in closed-loop robotic control pipelines, extending VLM utility beyond language tasks or reward coding to direct 3D trajectory synthesis. It quantifies how much VLM plan quality matters (keypoint vs. trajectory oracles ablation) and shows that few-shot prompting with execution feedback is a practical mechanism for correcting systematic VLM spatial errors (e.g., axis flipping). The work connects to the broader line of VLM-for-robotics research by showing that coarse, imprecise VLM outputs are sufficient when paired with residual RL — a key insight for designing VLM-grounded control systems that tolerate planning noise.