Skip to content

SeededGrasp: Language-Guided Grasping in Complex Scenes with Multiple Embodiments

🕒 Published (v1): 2026-07-22 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SeededGrasp decouples VLM-based semantic grounding from geometric grasp generation by having the VLM predict a 2D seed point that conditions a lightweight flow-matching model to synthesize grasp poses. This modular design avoids expensive end-to-end VLM fine-tuning while supporting multiple gripper embodiments in cluttered scenes. The system achieves 72% simulation and 78% real-world grasping success.

Problem

Existing language-conditioned grasping approaches either (a) use VLMs to predict grasp poses directly, sacrificing 3D spatial precision, or (b) train VLMs end-to-end with grasping models, requiring large language-annotated datasets and heavy compute. Neither approach scales to multi-embodiment grasping in cluttered, occluded scenes. Additionally, no prior dataset simultaneously provides cluttered multi-object scenes with multi-embodiment grasp annotations.

Method

SeededGrasp operates in two stages. First, a frozen VLM (Gemini 3.1 Flash) zero-shot predicts a 2D seed point on a bird's-eye-view (BEV) image, which is projected onto the 3D scene point cloud to localize the grasp target. Second, a flow-matching model conditioned on this seed point synthesizes the full grasp pose \(g = (T, R, \theta)\), where \(T \in \mathbb{R}^3\), \(R \in SO(3)\) (Euler angles), and \(\theta\) are gripper joint angles (padded to 16 DOF for Allegro).

Point clouds (scene: \(p_c \in \mathbb{R}^{2048 \times 3}\), gripper: \(p_r \in \mathbb{R}^{1024 \times 3}\)) are Fourier-encoded with local normals and curvature, then processed by GCN encoders using 16-nearest-neighbor graphs. Gripper identity is encoded via both a robot point cloud and a learnable query vector \(q_r \in \mathbb{R}^{512}\). A Diffusion Transformer (DiT) with cross-attention generates grasp poses via conditional flow matching, with loss:

\[\mathcal{L}_{cfm} = \mathbb{E}_{t \sim \beta(1.5,1.0),\,(g_t, f_r, f_c) \sim U}\left[|u_\theta(t, g_t, f_r, f_c) - u_t(t, g_t, f_r, f_c)|\right]\]

The combined loss \(\mathcal{L} = \lambda_\text{trans}\mathcal{L}_\text{trans} + \lambda_\text{rot}\mathcal{L}_\text{rot} + \lambda_\text{joints}\mathcal{L}_\text{joints}\) uses \(\lambda = (0.4, 0.04, 0.8)\). Classifier-free guidance (dropout of \(f_r, f_c\) during training; guidance weight \(w=1.1\) at inference) strengthens conditioning adherence. Inference uses Forward Euler integration with step size 0.2, initialized from a canonical grasp pose near the seed point rather than pure noise.

A new dataset of 2.56M grasps across 610 cluttered scenes, 334 objects, and 3 grippers (Franka Panda, Robotiq 3-Finger, Allegro) is generated by transforming mid-air grasps from MultiGripperGrasp (MGG) into valid in-scene poses via heuristic collision checks.

Key Contributions

  • Seed-point interface: VLM predicts a 3D contact-region hint, decoupling semantic reasoning from geometric execution without VLM fine-tuning or language-annotated grasp data.
  • Flow-matching grasp generation model conditioned on seed point, supporting three gripper embodiments through a shared architecture with per-gripper learnable query vectors.
  • First multi-embodiment tabletop grasping dataset in cluttered scenes: 2.56M poses across 3 grippers, 610 scenes, 334 objects (released publicly).

Results

  • Seed quality vs. DGN2.0 (Allegro only): SeededGrasp + Graspness seeds: 66.67% vs. DGN2.0 + Graspness seeds: 53.15%; SeededGrasp + VLM seeds: 72.97% vs. DGN2.0 + VLM seeds: 66.22%.
  • Multi-embodiment vs. GeoMatch: SeededGrasp achieves 71.38% / 72.16% / 72.07% (Franka/Robotiq/Allegro) vs. GeoMatch 25.35% / 35.71% / 37.97% (~35 pp gain).
  • Language-conditioned grasping vs. ShapeGrasp and GraspMAS (226 scene-prompt pairs): Object+part recognition 89.6% (easy) / 80.6% (hard) vs. ShapeGrasp 57.5% / 51.6% and GraspMAS 42.5% / 29.0%; overall grasp success 58.2% vs. 31.1% and 24.5%.
  • VLM ablation: Gemini 3.1 Flash 71.87% > Qwen 3.5 Flash 54.42% > GPT 5.4 42.61%.
  • Real-world (Delto 3-finger, 15 unseen objects, 10 trials each, 4 distractors): 78% overall success rate with zero-shot sim-to-real transfer (model trained on Robotiq, mapped via constant joint offsets).

Limitations

  • No arm kinematics awareness: predicted grasp poses may be unreachable given arm workspace constraints; joint planning and grasp selection are not co-optimized.
  • Single-view BEV conditioning for VLM seed prediction; occluded grasp regions in cluttered scenes may degrade seed quality.
  • Embodiment-specific learnable query vectors do not generalize zero-shot to unseen gripper morphologies; a unified joint-space representation is needed.
  • Training data over-represents power grasps from MGG, limiting performance on flat objects flush against surfaces.
  • Dataset skewed toward Franka Panda (1.61M train grasps vs. 246K Robotiq / 161K Allegro), causing disproportionate degradation for dexterous grippers under data reduction.

Relevance to Vision-Language Models

SeededGrasp demonstrates a practical pattern for injecting VLM spatial reasoning into a downstream geometric model without fine-tuning: the VLM acts as a zero-shot point-localization module, and its output (a 2D coordinate projected to 3D) serves as a compact, differentiable-free conditioning signal. The finding that Gemini 3.1 Flash substantially outperforms GPT-5.4 and Qwen 3.5 Flash on this spatial localization task (+17–29 pp) offers a concrete, task-specific VLM benchmark relevant to embodied VLM evaluation. For researchers tracking VLMs, this work exemplifies the "VLM as perception oracle, specialist model as executor" architectural pattern and quantifies the performance ceiling imposed by current VLM spatial reasoning capabilities in cluttered 3D scenes.