Skip to content

RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RobotSmith is an automated pipeline that uses collaborative VLM agents to design task-specific robotic tools from scratch, then jointly optimizes tool geometry and manipulation trajectories via physics simulation. It addresses the gap between human-designed tools (ill-suited for robotic manipulators) and the need for robot-appropriate tool creation. The system achieves 50% average success rate across 9 manipulation tasks, versus 21.4% for text-to-3D generation and 11.1% for retrieval baselines.

Problem

Existing robotic manipulation pipelines retrieve or assume human-designed tools that are often physically incompatible with robotic grippers. Template-based tool design methods generalize poorly across task types, while generic 3D generative models optimize for visual realism rather than functional utility. No prior work provides a fully automated, task-agnostic pipeline that co-designs tools and their usage plans from scratch.

Method

RobotSmith has three components operating in sequence:

  1. Critic Tool Designer: A dual-agent VLM loop (Proposer + Critic, both o3-mini) iteratively generates tool geometry in a parameterized, part-based JSON representation inspired by Constructive Solid Geometry (CSG). Primitives (box, cylinder, sphere) and text-to-3D generated components are assembled via a tree of geometric transformations. The Critic checks structural validity (connectivity, graspability, functional fit) and feeds back to the Proposer until it emits "Done."

  2. Tool Use Planner: A third VLM agent generates a manipulation trajectory as a sequence of three high-level APIs: grasp(obj, euler), move(pos, euler), and release(). Grasps are realized via sampling-based search; moves via IK + motion planning.

  3. Joint Optimizer: CMA-ES (population λ=20, 50 iterations) jointly optimizes tool shape parameters s (scaling each dimension in [0.5×, 2.0×] of initial) and trajectory waypoints q (±0.2 m translation, ±π rotation) against a task-specific scalar reward evaluated in the Genesis physics simulator.

Key Contributions

  • Unified automated pipeline combining VLM semantic reasoning with physics-based joint optimization for tool design and tool-use planning.
  • Parameterized, modular CSG-like tool representation enabling precise iterative refinement by LLM/VLM agents and gradient-free optimization.
  • Evaluation across 9 tasks spanning rigid, deformable, and fluid objects in simulation, plus real-world validation via 3D-printed tools on an XArm7 robot.
  • Demonstration of long-horizon autonomous task completion (sesame pancake making) using sequentially designed, task-specific tools.

Results

  • Overall success rate: Ours 50.0% vs. 3D generation (Meshy) 21.4% vs. retrieval 11.1% vs. no-tool 2.8%.
  • Best performance score (Pbest): Ours 0.94 vs. Meshy 0.72 vs. retrieval 0.53 vs. no-tool 0.24.
  • Task-level highlights (SR):
  • Lift a Piggy: 87.5% (ours) vs. 25.0% (retrieval, Meshy)
  • Hold a Phone: 75.0% vs. 67.5% (Meshy)
  • Flatten Dough: 62.5% vs. 12.5% (Meshy)
  • Reach: 50.0% vs. 0% (retrieval, no-tool)
  • Ablation (overall): removing trajectory optimizer drops SR from 50% to 5% (Pbest 0.94→0.28); removing tool optimizer drops SR to 32% (Pbest 0.67); removing text-to-3D drops SR to 33% (Pbest 0.69).

Limitations

  • Joint optimization currently restricted to scaling, rotation, and translation; topology changes and part reconfiguration are not supported, limiting structural diversity.
  • Misalignment between the Designer's intent and Meshy's uncontrolled output (size, orientation) causes design failures.
  • Grasp API fails on heavy tools or abrupt motions; trajectory optimizer struggles on tools with many shape parameters or long trajectories.
  • Tool orientation from the Designer is often underspecified (generic 90°/180° rotations), complicating fine manipulation (scooping, precise alignment).
  • Real-world validation covers only two tasks with 3D-printed tools; broader physical generalization is unverified.

Relevance to Agentic AI / LLM Agents

RobotSmith is a concrete instantiation of multi-agent LLM collaboration for embodied physical reasoning: a Proposer-Critic design loop, a Tool User planner, and a physics-based verifier jointly tackle a task that no single model could solve alone. It exemplifies how agentic pipelines can close the loop between language-level reasoning (tool concept) and physical-world grounding (geometry + trajectory optimization), a central challenge for embodied agents. The iterative Critic-Proposer loop with visual feedback is a direct analogue of reflection/self-refinement patterns widely studied in LLM agent research, here applied to artifact generation rather than text. The joint optimization over tool design and action planning is relevant to any agent architecture that must co-adapt its tools and policies in deployment.