Skip to content

SoFar: Language-Grounded Orientation Bridges Spatial Reasoning and Object Manipulation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SoFar introduces semantic orientation—a language-grounded, reference-frame-free representation that maps natural-language phrases (e.g., "plug-in direction of a USB") to unit vectors on 3D objects. The authors build OrienText300K (350K+ 3D models with GPT-4o-generated orientation-text pairs), train PointSO (a cross-modal 3D Transformer), and integrate both into the SoFar VLM-agent framework to enable 6-DoF spatial reasoning and robotic manipulation without task-specific fine-tuning.

Problem

VLMs used for robot manipulation planning can localize objects but are blind to object orientation, which is essential for 6-DoF manipulation (e.g., aligning a knife's blade, righting a wine glass, plugging in a USB). Prior pose-estimation methods rely on category-specific templates or fixed reference frames, limiting open-world, language-guided generalization. No large-scale dataset links free-text orientation descriptions to 3D geometry.

Method

Semantic Orientation is defined as a unit vector \(s^\mathcal{X}_\ell \in \mathcal{S}^{(2)}\) that maps an open-vocabulary phrase \(\ell\) (part names, interaction verbs, cardinal directions) to a geometric direction on a given 3D object \(\mathcal{X}\).

OrienText300K is constructed from Objaverse (~800K raw models): GPT-4o filters samples against six quality criteria (standard orthogonal views, clean backgrounds, meaningful geometry, etc.) achieving 88.3% filtering accuracy, then annotates semantic-view pairs at 97.1% accuracy, yielding 350K+ objects and 8M rendered images.

PointSO is a plain Transformer processing point clouds via FPS + KNN + PointNet feature extraction. CLIP text embeddings are injected into every Transformer layer via token-wise addition, and an MLP [CLS] head predicts the orientation vector. Training minimizes negative cosine similarity between predicted and ground-truth orientations.

SoFar system (pipeline): (1) A VLM identifies task-relevant object phrases and their semantic orientation descriptors from the RGB image + instruction; (2) Florence-2 + SAM segment objects to 3D point clouds; (3) PointSO estimates semantic orientations; (4) A 6-DoF scene graph \(G=(V,E)\) encodes per-object position, bounding box, and orientation set; (5) The VLM performs chain-of-thought reasoning over the scene graph to predict target poses; (6) Translation and rotation are computed analytically (Kabsch-Umeyama algorithm), and OMPL generates collision-free trajectories.

Key Contributions

  • Semantic orientation concept: open-vocabulary, template-free, reference-frame-free orientation representation linked to natural language.
  • OrienText300K: 350K+ 3D objects with diverse language-grounded orientation annotations; 8M rendered images; GPT-4o-automated construction with quality validation.
  • PointSO: cross-modal 3D Transformer with CLIP text fusion for zero-shot semantic orientation prediction across unseen objects.
  • SoFar framework: integrates PointSO with SAM/Florence-2 and a VLM into a modular 6-DoF reasoning and manipulation pipeline; generalizes across gripper, suction-cup, and dexterous-hand embodiments.
  • Open6DOR V2: extended benchmark adding closed-loop execution tasks in robosuite/LIBERO environments.
  • 6-DoF SpatialBench: 223-sample VQA benchmark with quantitative positional and orientational annotations.

Results

  • Open6DOR V1 (perception, Isaac Sim): SoFar achieves 93.0% overall on Position track and 57.0% on Rotation track (6-DoF overall: 48.7%), vs. Open6DOR-GPT at 74.9% / 48.6% and GPT-4V at 45.2% / 9.2%. Planning time: 8.5 s vs. 358.3 s for GPT-4V.
  • Open6DOR V2 (execution, LIBERO): SoFar 63.7% Position / 25.7% Rotation / 18.3% 6-DoF overall, vs. OpenVLA 46.2% / 17.6% / 18.5%.
  • SIMPLER-Env Google Robot (zero-shot): SoFar 0.907 average on Variant Aggregation and 0.749 on Visual Matching, outperforming RT-2-X (0.661 / 0.606) trained on OXE.
  • SIMPLER-Env WidowX (zero-shot): SoFar 0.583 average vs. SpatialVLA (Bridge-trained) 0.427.
  • PointSO accuracy (OrienText300K val, 128 objects): PointSO-L achieves 81.25% at 45°, 72.66% at 15°, 65.63% at 5°; robust under single-view (76.56%), Gaussian jitter (81.25%), random rotation (77.34%), and combined corruption (74.22%) at 45°.
  • 6-DoF SpatialBench: SoFar 43.9% total vs. GPT-4o 36.2%, SpatialBot 32.7%; >18% absolute improvement over best baseline.
  • Real-world 60-task suite: SoFar outperforms CoPa and ReKep-Auto across all tracks (position-simple through 6-DoF comprehensive).

Limitations

  • Decoupled pipeline propagates sub-module errors: perception failures (inaccurate segmentation, unstable grasps) cascade to manipulation failure.
  • Objects can be placed in unexpected poses due to rotation induced during execution (e.g., pen placement).
  • 6-DoF SpatialBench is small (223 samples), limiting statistical power.
  • Open6DOR V2 execution success rates remain relatively low (~18–26% on 6-DoF track), indicating that vanilla trajectory planning is insufficient for fine-grained manipulation; prehensile grasping strategies are needed.
  • PointSO trained on Objaverse; sim-to-real domain gap not fully characterized.

Relevance to Vision-Language Models

SoFar directly extends the VLM spatial-reasoning line (SpatialVLM, SpatialRGBT, SpatialBot) by identifying orientation understanding as a missing modality that VLMs cannot produce natively and providing a plug-in module (PointSO + scene graph) to supply it. The work demonstrates a concrete pattern for augmenting frozen VLMs with task-specific geometric perception without fine-tuning, which is broadly applicable wherever VLMs serve as planning agents over physical scenes. The GPT-4o-driven dataset construction pipeline—using VLMs as discriminators rather than generators of geometric values—offers a scalable methodology for other grounded representation problems. For researchers tracking VLM embodiment and grounding, SoFar highlights that 6-DoF action generation requires explicit orientation representations beyond what current VLMs learn from image-text pretraining alone.