VLBiMan: Vision-Language Anchored One-Shot Demonstration Enables Generalizable Bimanual Robotic Manipulation¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; VL anchoring enables one-shot generalization for bimanual manipulation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

VLBiMan is a training-free bimanual robotic manipulation framework that generalizes from a single human demonstration by decomposing it into reusable invariant skill primitives and scene-adapting variable components via VLM-grounded object anchors. It avoids policy retraining by separating what to achieve (task-invariant sub-skills) from how to execute (scene-adaptive trajectory adaptation). Experiments across ten bimanual tasks show substantial success-rate gains over imitation and zero-shot baselines.

Problem¶

Imitation-learning approaches (VLAs, ALOHA-family, RDT-1B) require large demonstration sets and full retraining for each new object/task variant, while zero-shot modular methods (ReKep, MOKA) rely on brittle prompt engineering and unreliable trajectory execution. No prior method efficiently handles bimanual generalization from a single demonstration across novel object instances, spatial rearrangements, and cross-embodiment transfer without retraining.

Method¶

VLBiMan operates in three stages on a single labeled demonstration \(\mathcal{D} = \{(O_t, A_t)\}_{t=1}^{T}\):

Task-Aware Bimanual Decomposition: Keypose-driven segmentation splits \(\mathcal{D}\) into atomic primitives via velocity/acceleration discontinuities and gripper state transitions. Each primitive \(M_i\) is labeled invariant if \(\forall t \in \tau_i,\ \text{bind}(o_k, r, t)=1\) and \(\text{geometry}(o_k)\approx\text{geometry}(o_k^{\text{demo}})\) within tolerance \(\epsilon_g\), otherwise variable.
Vision-Language Anchored Adaptation: Florence-2 and SAM2 segment task-relevant objects from new scene observations. For variable primitives, adaptation computes (a) a relative position shift \(\Delta x = p_{\text{new}} - p_{\text{demo}}\) from stereo-back-projected representative points (mask centroid or table-contact boundary), and (b) a relative orientation \(\Delta\theta = \angle(v_{\text{new}}, v_{\text{demo}})\) from second-order image moment principal axes. Category-level shape changes (\(\Delta h_k\) from point cloud z-extent) adjust gripper spacing and vertical placement.
Autonomous Trajectory Composition: Adapted variable and reused invariant modules are concatenated along the original temporal structure. Progressive IK refinement iterates \(q^{(n+1)} = \text{IK}(T_g^{(n)})\) with spline-interpolated waypoints to handle displacement, and dynamic collision compensation adds proximal/vertical offsets \(\tilde{x}_{\text{goal}} = x_{\text{goal}} + \delta_{\text{base}} u_\| + \delta_z u_z\).

Human-in-the-loop waypoint refinement is done once per task during setup; subsequent scene variations require no human input.

Key Contributions¶

Training-free one-shot bimanual manipulation framework that generalizes to novel object instances, spatial rearrangements, and tool-use tasks without retraining.
Task-aware decomposition distinguishing invariant from variable motion primitives using object-robot coupling criteria.
VLM-anchored adaptation pipeline (Florence-2 + SAM2) that grounds object identity and computes geometric offsets without CAD models or 6-DoF pose estimation.
Compositional generalization via atomic skill splicing across long-horizon multi-stage tasks.
Cross-embodiment transfer: skills from human demonstrations directly instantiated on different robot platforms.

Results¶

All tasks use 25 trials; objects randomly placed/replaced each trial.

Six basic bimanual tasks, new placements + same objects, no interference: VLBiMan 85.3% vs. ReKep+ 43.3%, ReKep 36.0%, Robot-ABC 44.7%, MAGIC 26.7%, Mechanisms 13.3%.
Six basic tasks, new placements + same objects, with interference: VLBiMan 69.3% vs. ReKep+ 22.7%, ReKep 18.0%, Robot-ABC 24.7%, MAGIC 12.0%, Mechanisms 4.0%.
Six basic tasks, new placements + novel instances, no interference: VLBiMan 78.0% vs. ReKep+ 29.3%, ReKep 24.0%, Robot-ABC 27.3%, MAGIC 12.7%.
Six basic tasks, novel instances, with interference: VLBiMan 59.3% vs. ReKep+ 15.3%, ReKep 7.3%, Robot-ABC 12.0%, MAGIC 4.0%.
Table 2 (long-horizon tasks) results present but truncated in provided text; qualitative claims indicate VLBiMan maintains "outstanding performance and significant advantages."
Drastic reduction in demonstration requirements vs. all imitation baselines.

Limitations¶

Human-in-the-loop waypoint refinement is required once per task during initial setup, introducing manual overhead not present in fully automated pipelines.
Invariance criterion depends on object geometry tolerance \(\epsilon_g\); highly deformable or articulated objects violate the rigid-coupling assumption and fall outside the framework's scope.
Adaptation relies on stereo camera calibration and hand-eye transformation accuracy; calibration errors propagate directly to pose offsets.
One-time physical replay is needed to detect and correct collisions after novel category substitution, limiting full autonomy.
Cross-embodiment transfer is demonstrated but the scope of robot platforms tested is not specified in the provided text.

Relevance to Vision-Language Models¶

VLBiMan demonstrates a concrete deployment pattern for VLMs in closed-loop robotic control: rather than using VLMs for end-to-end policy generation, they serve as robust perception modules (Florence-2 for grounding, SAM2 for segmentation) that anchor geometric adaptation without requiring fine-tuning or prompt engineering at execution time. This modular VLM-as-perception paradigm—where semantic understanding is decoupled from motion synthesis—is directly relevant to the broader effort to make VLMs actionable in physical settings. The work also highlights a key failure mode of purely VLM-driven zero-shot planners (e.g., ReKep, GPT-4o-based pipelines): unreliable trajectory execution due to ambiguous intermediate representations, which VLBiMan sidesteps by conditioning on a labeled demonstration. For VLM researchers, this positions demonstration-conditioned VLM grounding as a more robust alternative to prompt-only zero-shot manipulation.