Skip to content

From Seeing to Doing: Bridging Reasoning and Decision for Robotic Manipulation

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; bridging reasoning to action for novel robotic manipulation scenarios

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FSD (From Seeing to Doing) is a VLM-based framework for robotic manipulation that generates embodiment-agnostic visual aids—spatial affordance boxes, affordance points, and visual traces—through a structured spatial relationship-focused chain-of-thought (SrCoT) reasoning mechanism. Rather than mapping directly from pixels to actions, FSD first builds a spatial relationship graph over scene objects, then derives visual aids as intermediate representations for downstream control. It achieves 40.6% zero-shot success in SimplerEnv simulation and 72% across 8 real-world tasks, outperforming the strongest baseline by 30%.

Problem

Existing VLA models inherit VLM backbones but fail at zero-shot generalization due to two compounding issues: (1) data scarcity—embodied datasets are orders of magnitude smaller than internet-scale vision-language corpora, preventing scaling-law-like generalization; and (2) embodiment heterogeneity—diverse action spaces across robot platforms make end-to-end vision-to-action mapping prone to overfitting and knowledge forgetting. Prior affordance-based methods predict raw coordinates without explicit reasoning, hindering semantic grounding of instructions to correct objects.

Method

FSD has three components:

1. Spatial Relationship-Focused Visual Chain-of-Thought (SrCoT).
Given an image \(X_v\) and task instruction \(X_q\), SrCoT first produces a <Description> block containing object-centric region captions and a spatial relationship graph where nodes are objects with normalized coordinates \((p,q)\in[0,999]^2\) and edges encode relations (left of, above, behind, etc.). A <Reasoning> block then performs multi-hop analysis—anchoring start/end points via object references, iteratively deriving intermediate waypoints with explicit logical links—to produce visual aids: bounding boxes \(B=[x_1,y_1,x_2,y_2]\), affordance points \(P=\{(x_i,y_i)\}\), or visual traces \(\tau=\{x_t\}_{t=1}^{T}\) (fixed \(T=8\) for simplification). Coordinates are treated as text integers.

2. Hierarchical weak-to-strong data pipeline.
A 300K SFT dataset is constructed from BridgeDataV2, RT-X, and Droid across five capability levels: (1) region grounding, (2) spatial relationship understanding via 3D scene graphs reconstructed with Metric3Dv2 + WildCamera, (3) multi-hop spatial reasoning Q&A, (4) spatial affordance labels extracted from terminal demonstration frames, (5) visual trace labels via self-supervised keypoint extraction + CoTracker projection. Strict rule-based filtering and iterative refinement are applied throughout.

3. Self-consistency alignment.
To align coordinate space with image semantics (absent from pretraining), FSD trains bidirectionally: forward \((X_v, X_q)\to\tau\) and inverse \((X_v,\tau)\to X_q\). This forces the model to comprehend the physical meaning of generated coordinates.

Architecture: Frozen CLIP-ViT-L encoder + trainable linear projector + Vicuna-13B LLM, initialized from ASMv2, following the LLaVA-1.5 instruction tuning pipeline. Stage 1 trains on Level 1–3 data mixed with 1.4M general VQA data to prevent catastrophic forgetting; Stage 2 trains visual aids generation with self-consistency on Level 4–5 data.

Action execution: 2D visual traces are back-projected to 3D via pinhole camera model with depth input, grasp pose is matched via GraspNet, and trajectory is optimized in SE(3) via gradient-descent interpolation. Affordance-only tasks use CuRobo as motion planner.

Key Contributions

  • SrCoT mechanism: spatial relationship graphs as reasoning anchors for structured visual-spatial CoT, applicable beyond embodied domains.
  • Self-consistency alignment: bidirectional forward/inverse training to ground coordinate spaces in image semantics.
  • Hierarchical 300K dataset: 5-level capability pipeline covering 10+ embodiments, constructed automatically from large-scale embodied datasets.
  • VABench: 300 manually annotated real-world problems evaluating spatial affordance (point accuracy) and visual trace generation (MAE, RMSE, LLM score 1–10).
  • State-of-the-art performance: best open-source rank across 8 spatial reasoning/reference benchmarks; 40.6% zero-shot SimplerEnv; 72% real-world success (+30% over RoboPoint).

Results

General spatial reasoning (5 benchmarks, Table 1, 13B open-source models): - FSD average rank: 1.3 vs. SAT-Dynamic 2.8, RoboPoint 2.8, ASMv2 3.1 - CVBench avg: FSD 80.9 vs. RoboPoint 68.2, ASMv2 66.4 - EmbSpatial-Bench: FSD 63.3 vs. RoboPoint 51.4, ASMv2 57.4

Object/free-space reference (Table 2): - RoboRefIt: FSD 56.7 vs. RoboPoint 49.8, ASMv2 48.4, GPT-4o 15.3 - Where2Place: FSD 45.8 vs. RoboPoint 46.0, ASMv2 22.0

VABench visual aids generation (Table 3): - VABench-Point accuracy: FSD 61.82% vs. RoboPoint 19.09%, ASMv2 10.07%, GPT-4o 9.30% - FSD w/o SrCoT drops to 26.21%; w/o alignment drops to 55.92%—validating both components.

Zero-shot robot manipulation: - SimplerEnv simulation: FSD 40.6% success rate - Real-world xArm (8 tasks): FSD 72% success rate, +30% over strongest baseline

Limitations

  • Visual trace generation is simplified to a fixed 8-point sequence, which may be insufficient for high-DOF or highly curved trajectories.
  • 3D back-projection requires a depth camera; performance degrades in depth-unavailable or poorly calibrated settings.
  • VABench is small (300 problems), hand-annotated from a narrow set of sources; generalization of benchmark conclusions is uncertain.
  • Backbone is Vicuna-13B (2023-era LLM), which may be outpaced by newer LLM backbones for reasoning quality.
  • Depth-based grasp pose estimation via GraspNet introduces an additional failure mode not ablated in the paper.
  • Real-world evaluation covers only 8 tasks on a single xArm platform; cross-embodiment real-robot generalization is not demonstrated.

Relevance to Vision-Language Models

FSD exemplifies a growing paradigm of using VLMs not as end-to-end action predictors but as structured intermediate reasoners that produce grounded spatial representations—directly relevant to researchers studying VLM spatial reasoning, chain-of-thought grounding, and embodied perception. The SrCoT mechanism is a concrete instantiation of visual CoT that anchors language-level reasoning in coordinate-tagged object references, addressing the notorious coordinate hallucination problem in VLMs. The self-consistency alignment technique—training inverse tasks to ground coordinate semantics—is broadly applicable to any VLM task involving spatial or structured output generation. For VLM researchers, the weak-to-strong hierarchical data pipeline and the VABench benchmark offer a reusable evaluation and training framework for spatial visual aids, filling a gap left by benchmarks focused solely on natural-image spatial QA.