Skip to content

Learning Spatial-Aware Manipulation Ordering

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OrderMind is a unified framework for robot manipulation ordering in cluttered scenes that learns spatial priorities directly from RGB-D images via a k-NN spatial graph and attention-based priority scoring. Rather than invoking a VLM at inference time, it distills VLM knowledge into a compact (35–42M parameter) model that runs at 5.6–21.3 FPS. On a 163K-sample benchmark, it surpasses both heuristic pipelines and privileged VLMs (with ground-truth poses) in success rate.

Problem

Existing manipulation ordering pipelines either use brittle heuristic algorithms that fail under scene variation, or chain a detector with a VLM for ordering—achieving at most 0.2 FPS and requiring privileged ground-truth object information. Neither approach learns spatial dependencies end-to-end, and VLMs in the loop introduce latency incompatible with real-time control.

Method

OrderMind maps RGB-D images and end-effector pose to per-object priority scores in a single forward pass:

  1. Spatial Context Understanding (SCU): Represents each object as its 3D bounding box center with intrinsic (class, extent) and extrinsic (pose) attributes. A k-NN graph over object centers aggregates neighbor features via concatenation + linear + max-pool (PointNet-style), and separately encodes each object's transform relative to the end-effector pose.

  2. Temporal Priority Structuring (TPS): Object tokens from a vision backbone are refined via self-attention (inter-object), then cross-attended against both a global scene token (aggregated by max-pool) and raw visual features, yielding per-object priority scores.

  3. Preferential Order Alignment: Hungarian matching aligns predicted objects to ground-truth; a pairwise ranking loss (logistic with log-weighted emphasis on large rank gaps) trains continuous priority scores rather than discrete indices.

  4. Spatial Priors Order Labeling (SPOL): Qwen2.5-VL generates training labels guided by two geometric priors—an Independence Prior (objects with minimum horizontal distance ≥ threshold τ are labeled early) and a Local Optimality Prior (objects with no other object occupying their vertical clearance volume are labeled early). This removes the need for manual annotation.

Key Contributions

  • End-to-end spatial-aware manipulation ordering framework (no VLM at inference), achieving real-time performance.
  • k-NN spatial graph encoding both object–object and object–manipulator geometric relationships.
  • SPOL: VLM-guided label generation using Independence and Local Optimality spatial priors, enabling training without manual annotation.
  • Manipulation Ordering Benchmark: 163,222 RGB-D samples across easy/moderate/hard difficulty (24/36/60 objects per scene).
  • Preferential Order Alignment loss using Hungarian matching + pairwise logistic ranking with log-weighted margin.

Results

  • OrderMind (41.8M params, 5.6 FPS): 96.5% / 95.3% / 95.4% success rate (SR) on easy/moderate/hard simulation, outperforming the best privileged VLM (Gemini-2.5: 92.4% / 84.6% / 78.5%) and all two-stage pipelines (best: UniDet3D + InternVL3 at 38.3% / 42.9% / 43.1% SR).
  • OrderMind-Mini (35.2M params, 21.3 FPS): 94.2% / 89.6% / 90.4% SR—real-time capable while exceeding all baselines that lack privileged information.
  • Real-world closed-loop: 93.3% / 78.5% / 76.6% SR on easy/moderate/hard scenes.
  • Order stability (Levenshtein Distance at replanning interval=5): OrderMind LD=3.5 vs. YOLO11-seg+SPH LD=6.6.
  • Robustness to label noise: SR degrades from 85.4% to 79.9% at 20% noise; meaningful above 50% noise.
  • Ablation (moderate difficulty): Full model (SCU+TPS+SPOL) achieves 95.3% SR vs. 76.1% baseline; SPOL alone contributes the largest individual gain.

Limitations

  • Assumes a static scene during execution; cannot adapt to dynamic changes mid-task.
  • Relies on accurate 3D bounding-box estimation; performance degrades under severe occlusion where 3D attributes are unreliable.
  • 39% of real-world failures stem from incorrect ordering; 21% each from inaccurate 3D rotation and deformable objects with no suitable suction surface—indicating perception bottlenecks beyond ordering logic.
  • SPOL labels are generated by a VLM (Qwen2.5-VL); at 70% label noise the model degenerates to near-random performance, so label quality remains a dependency.

Relevance to Vision-Language Models

OrderMind exemplifies a growing paradigm of VLM-as-teacher / lightweight-model-as-student: rather than deploying a VLM at inference, it uses one (Qwen2.5-VL) to generate structured spatial annotations offline, then distills that knowledge into a real-time network. This is directly relevant to researchers tracking VLMs because it quantifies the inference-quality frontier of frontier VLMs (GPT-4o, Claude-3.7-Sonnet, Gemini-2.5) on fine-grained 3D spatial reasoning—and shows they plateau at ~78–92% SR even with privileged inputs. The SPOL pipeline demonstrates a reusable recipe for using VLM spatial priors to bootstrap annotation for robotics tasks where manual labeling is intractable, and the failure modes reveal where current VLMs remain weak: stacking-order inference and dense occlusion reasoning.