VLMgineer: Vision-Language Models as Robotic Toolsmiths¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; VLMs as robotic toolsmiths — tool design as measure of physical intelligence

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

VLMgineer is the first fully automatic framework that uses Vision-Language Models together with evolutionary search to jointly design novel physical tools (in URDF) and their accompanying action policies for robots, without any human-specified parametric templates or task-specific prompts. It introduces RobotToolBench, a 12-task simulation benchmark, and achieves large gains over human-specified and existing human-crafted tools, with successful sim-to-real transfer.

Problem¶

Prior robotic tool design methods require manual pre-definition of parametric design spaces, fixed trajectories, or task-specific templates, making them labor-intensive to scale. Existing large-model-guided evolutionary approaches for robot design optimize morphology only, leaving action policy as a separate sequential step and missing the tight coupling between tool geometry and manipulation strategy. Single-shot VLM prompting for tool design yields functional but non-specialized, non-optimized geometries.

Method¶

VLMgineer implements a three-phase VLM-guided evolutionary loop operating on raw environment source code, an environment image render, and a task description:

Population initialization: The VLM generates \(K\) candidate (tool, action) pairs. Tools are expressed as URDF modular blocks attached to the robot's end-effector link; actions are \(N \times 7\) waypoint arrays encoding 6-DoF end-effector poses and gripper open/close commands. Each iteration proposes \(n\) tool designs \(\times\) \(m\) action plans, yielding \(n \times m\) tool-action pairs.
Fitness evaluation: Each pair is evaluated in PyBullet simulation using task-specific normalized reward functions \(R: S \rightarrow r \in [0, 1]\). Top-\(k\) designs are retained.
In-context crossover and mutation (evolution): The retained elite designs are fed back to the VLM as in-context examples. The VLM is explicitly prompted to apply either mutation (change exactly one design aspect) or crossover (combine elements from two elite designs), guided by its learned inductive biases about what makes tools physically effective.

Joint tool-action sampling is the key architectural choice: rather than sequentially optimizing geometry then policy, simultaneous generation allows the VLM to navigate the joint design space toward the Pareto frontier, and appropriately shaped tools directly simplify required actions.

Key Contributions¶

VLMgineer: the first fully automatic framework for scratch-designed robotic tools and action co-design via VLM-guided evolutionary search, requiring no task-specific templates or human-in-the-loop steps.
RobotToolBench: a 12-task simulation benchmark (PyBullet, Franka Panda) explicitly designed to evaluate robotic tool and policy design methods, covering everyday and novel scenarios.
Empirical demonstration that evolutionary refinement elicits substantially greater physical creativity from VLMs than single-shot prompting alone.
Successful zero-shot sim-to-real transfer of co-designed tools and action policies to a physical robot.

Results¶

vs. Human Prompt baselines (robotics expert, LLM expert, layperson — no evolutionary search): VLMgineer achieves an average normalized improvement of 64.7% across all 12 RobotToolBench tasks; outperforms even the robotics-expert human on both peak and average reward.
vs. existing human-crafted tools (RLBench original tools on 4 adapted tasks): average normalized improvement of 24.3%; VLMgineer attains maximum possible reward on every RLBench-derived task with higher reliability.
Ablation — evolution vs. brute-force sampling (same budget: 8000 evaluations total; 4 iterations × 2000 samples vs. flat 8000 VLM samples): evolutionary search outperforms brute-force sampling by 119.2% on average across three tasks.
vs. Franka Gripper (no tool): the default gripper fails on the majority of the 12 tasks by design; VLMgineer solves them effectively.

Limitations¶

Action representation uses discrete waypoints rather than closed-loop controllers, limiting applicability to highly dynamic or contact-rich tasks (though the authors note the framework can warm-start closed-loop policies).
Evaluation depends on simulation-defined fitness functions; design of meaningful per-task reward functions still requires human effort.
Benchmark limited to 12 tasks with a single robot morphology (Franka Panda) in PyBullet; diversity and generalization to more complex environments is untested.
Evolutionary search with VLM inference is computationally expensive (4 iterations × 2000 samples); wall-clock cost and token cost are not quantified in the excerpt.
No analysis of failure modes or tasks where VLMgineer does not converge to a good design.

Relevance to Vision-Language Models¶

VLMgineer is a direct stress-test of whether pre-trained VLMs encode genuine physical and spatial reasoning, demonstrating that this capability can be unlocked through iterative evolutionary prompting far beyond what single-shot querying reveals. The use of URDF as a structured, code-like tool representation is an important design finding: it aligns with VLMs' strong code-generation priors, offering a transferable principle for grounding VLM generation in physically realizable outputs. The work contributes to the broader line of VLM-guided program synthesis and design (alongside Eureka for reward functions and EvoPrompting for algorithms), extending the frontier to physical artifact design with embodied evaluation. For VLM researchers, the 119.2% gain of evolutionary refinement over flat sampling quantifies how much iterative in-context feedback amplifies VLM physical creativity.