Skip to content

GROW\(^2\): Grounding Which and Where for Robot Tool Use

🕒 Published (v1): 2026-06-29 17:56 UTC · Source: Arxiv · link

Why this paper was selected

David Hsu; open-world affordance grounding for creative robot tool use with VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GROW\(^2\) decomposes open-world affordance grounding for robot tool use into two levels — semantic (VLM selects tool + task-relevant parts) and geometric (vision foundation models ground parts into 3D regions) — using object parts as an intermediate abstraction. It achieves zero-shot generalization to open-category objects and outperforms baselines on affordance benchmarks, simulation, and real-world tool use without requiring task-specific training data.

Problem

Robots that improvise tool use (e.g., cutting cake with a plate edge when no knife is available) must solve open-world affordance grounding: which object can serve as the tool and where the interaction regions are on that object. Prior work assumes the tool is pre-specified, relies on fixed tool-function mappings, or trains end-to-end models on large affordance datasets that struggle to generalize to unseen task-object pairs.

Method

GROW\(^2\) introduces object parts (blade, rim, handle, etc.) as an intermediate representation that captures both affordance semantics and geometric structure, splitting grounding into two levels:

  1. Tool and part selection (semantic level): A VLM enumerates objects and their parts from the RGB observation (with SAM\(^3\) segmentation to reduce hallucination). Given the task instruction, the VLM selects the tool object \(o_A\), target object \(o_B\), grasp part \(p_G\), tool-interaction part \(p_A\), and target-interaction part \(p_B\), using few-shot in-context learning with chain-of-thought examples.

  2. 3D affordance grounding (geometric level): For each selected part, the pipeline (a) reconstructs a high-fidelity object mesh from single-view RGB-D using SAM\(^3\)D conditioned on the depth-derived point map; (b) performs ICP-based rigid registration to align the mesh to the observed camera frame; (c) renders the registered mesh from \(K=8\) viewpoints; (d) segments the part description in each rendered view using SAM\(^3\) (with VLM fallback for complex queries); and (e) fuses the multi-view 2D masks into a 3D point set via back-projection + DBSCAN clustering, discarding outliers and retaining the largest cluster as the predicted affordance region.

Key Contributions

  • Part-based decomposition of affordance grounding into semantic (VLM) and geometric (vision foundation models) levels, eliminating dependence on large-scale affordance annotations
  • Zero-shot generalization to open-category objects and novel tool functions without task-specific training
  • Complete 3D affordance regions from single-view RGB-D, achieved by reconstructing the object mesh and rendering from multiple views to resolve occlusion and self-occlusion
  • New GROW\(^2\) Bench multi-object benchmark with 10 task types and 500 human-annotated instances for evaluating tool selection + affordance grounding jointly

Results

  • AGD20K (2D affordance): Best SIM (0.418) and NSS (1.334), competitive KLD (1.506) — outperforms UAD, AffordanceLLM, LOCATE, Cross-View-AG
  • PIAD (3D affordance): Best aIoU (9.0), AUC (74.2), tied lowest MAE (0.102) — outperforms GEAL, LASO, IAGNet, XMF, PFusion
  • GROW\(^2\) Bench (multi-object): Highest IoU across all 10 task types (average ~0.48) vs DINOv3 (~0.18), SD-DINO (~0.22), MOKA (~0.33)
  • Simulation (5 task types, 100 scenes): 86.7% average success vs next-best 62.7% (GROW\(^2\) w/o multi-view) and 47% (UAD given ground-truth tool); ablated multi-view variant drops ~24 points, confirming its importance
  • Real-world (Franka arm, 50 scenes): 78.0% average success vs 44.0% for MOKA

Limitations

  • VLM hallucination in tool/part selection accounts for the largest share of failures — the single-shot VLM call lacks iterative scene inspection
  • High latency (~16.6 s per scene) from sequential VLM and vision foundation model inference, limiting dynamic manipulation
  • Relies on multiple vision foundation models (SAM\(^3\), SAM\(^3\)D, VLM) chained sequentially, creating pipeline brittleness
  • Failure cases also arise from inaccurate mesh reconstruction scale/geometry and SAM\(^3\) producing invalid masks for certain part descriptions

Relevance to Vision-Language Models

This paper is a direct application of VLMs to a core robotics problem: using VLM commonsense reasoning to bridge the gap between language instructions and physical affordances. The part-based decomposition is notable because it uses VLMs only for semantic selection (where they excel) and delegates precise geometric grounding to specialized vision models, avoiding the spatial hallucination issues that plague end-to-end VLM affordance prediction (e.g., MOKA). For the VLM community, GROW\(^2\) demonstrates that VLMs can serve as effective semantic routers in embodied systems when paired with non-parametric geometric backends.