Skip to content

RobotSmith: Generative Robotic Tool Design for Acquisition of Complex Manipulation Skills

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

RobotSmith is an automated pipeline that uses collaborative VLM agents to design task-specific robotic tools from scratch, then jointly optimizes tool geometry and manipulation trajectories via physics simulation. It achieves a 50.0% average success rate across 9 manipulation tasks involving rigid, deformable, and fluid objects, compared to 21.4% for 3D generation and 11.1% for retrieval baselines.

Problem

Existing robotic manipulation pipelines assume tools are either retrieved from human-designed asset libraries (poorly suited for robotic grippers) or generated with generic 3D models (not optimized for functional utility). Template-based tool design methods cannot generalize beyond narrow tool families, and latent-space 3D generation lacks task-grounded functional reasoning. No prior framework jointly addresses tool design and tool use in a unified, automated manner.

Method

RobotSmith has three modules operating in sequence:

  1. Critic Tool Designer: Two VLM agents (Proposer and Critic, both o3-mini-based) engage in iterative design refinement. The Proposer outputs tool geometry as JSON specifying primitive parts (box, cylinder, sphere) and a CSG-like assembly tree. Multi-view renders are passed to the Critic, which checks structural validity (connectivity, graspability, reach) and returns feedback until issuing a termination signal.

  2. Tool Use Planner: A Tool User VLM agent generates a robot trajectory as a sequence of three high-level API calls — grasp(obj, euler), release(), and move(pos, euler) — abstracting away IK and motion planning.

  3. Joint Optimizer: CMA-ES (population λ=20, 50 iterations) jointly optimizes tool shape parameters s (primitive dimensions, ±50–100% of initial value) and trajectory parameters q (Cartesian waypoints ±0.2 m, rotations ±π) against a task-specific scalar reward evaluated in the Genesis physics simulator. A text-to-3D model (Meshy) augments primitives for geometrically complex parts.

Key Contributions

  • Unified framework coupling VLM-based collaborative design with physics-based joint optimization for tool geometry and trajectory.
  • Modular, parameterized, CSG-inspired tool representation enabling interpretable refinement and downstream gradient-free optimization.
  • Demonstration on 9 diverse tasks (rigid, deformable, fluid) in simulation plus real-world validation via 3D-printed tools on an XArm7 robot.
  • Long-horizon validation: autonomous pancake-making sequencing flatten, scoop, spread, and sprinkle subtasks with distinct generated tools.

Results

  • Overall (9 tasks): Ours 0.94 Pbest / 50.0% SR vs. Meshy 0.72 / 21.4%, Retrieval 0.53 / 11.1%, No Tool 0.24 / 2.8%.
  • Reach: Ours 0.93 / 50.0% vs. Meshy 0.69 / 0.0%, Retrieval 0.09 / 0.0%.
  • Lift a Piggy: Ours 1.00 / 87.5% vs. Meshy 1.00 / 25.0%, Retrieval 1.00 / 25.0%.
  • Flatten Dough: Ours 1.00 / 62.5% vs. Meshy 1.00 / 12.5%.
  • Ablations: removing the trajectory optimizer collapses SR from 50% to 5% (most critical component); removing tool geometry optimizer drops to 32%; removing text-to-3D drops to 33%.

Limitations

  • Joint optimizer is restricted to scaling, rotation, and translation of fixed-topology parts; topology changes (part addition/removal, part reconfiguration) are not supported.
  • Meshy text-to-3D outputs are not fully controllable in size or orientation, causing misalignment with the Designer's specification.
  • Grasp API fails for heavy tools or abrupt required motions.
  • CMA-ES struggles when tools have many shape parameters or trajectories are long and complex.
  • Evaluation is limited to 9 curated tasks; real-world validation covers only 2 tasks (dough calabash and hold a phone).

Relevance to Vision-Language Models

RobotSmith is a direct application of VLMs as embodied physical reasoners: it exploits their spatial reasoning, affordance perception, and commonsense physical inference to drive iterative tool design without domain-specific training data — a use case beyond the typical VLM evaluation of visual QA or captioning. The dual-agent (Proposer/Critic) architecture demonstrates a collaborative multi-agent VLM pattern for structured generation, relevant to researchers studying VLM agentic frameworks. The key insight — that VLMs provide strong geometric priors but insufficient precision for physical execution, requiring simulation-based refinement — is a general lesson for any VLM-in-the-loop robotic pipeline. This connects to the broader trend of VLMs as planning and design oracles in embodied AI, complementing reward modeling and code-generation applications.