Skip to content

Generating CAD Code with Vision-Language Models for 3D Designs

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CADCodeVerify is an automated iterative refinement pipeline that uses a VLM to generate binary verification questions about a rendered 3D CAD object, answer them from multi-angle images, and produce corrective feedback — closing the loop without human intervention. The authors also release CADPrompt, the first benchmark (200 natural-language → CAD-code pairs) for this task. Applied to GPT-4, the method achieves a 7.30% reduction in Point Cloud distance and a 5.5% compile-rate gain over the prior state-of-the-art.

Problem

LLM/VLM-generated CAD scripting code (e.g., CADQuery Python) frequently produces 3D objects that deviate structurally or geometrically from the user specification, and current refinement methods either require costly human-in-the-loop feedback or access to ground-truth geometry (geometric solvers). There is no quantitative benchmark for evaluating CAD code generation quality.

Method

The pipeline has three stages: 1. Code Generation — VLM generates CADQuery Python from a natural-language prompt (zero-shot or few-shot). 2. Code Execution — CADQuery compiler renders the code to an STL mesh; syntax errors are fed back to the VLM for repair until compile succeeds or iteration limit is reached. 3. Code Refinement via CADCodeVerify — A two-step loop: (a) Question-Answer Generation: the VLM generates 2–5 binary (Yes/No/Unclear) verification questions about the prompt, then answers them using four reference renders (0°/90°/180°/270°) with Chain-of-Thought reasoning; (b) Feedback Generation: negative answers are synthesized into actionable corrective feedback, which is fed back into the VLM to revise the code. The loop repeats for M iterations; if all answers are "Yes", refinement terminates early.

Key Contributions

  • CADCodeVerify: fully automated, human-free, model-agnostic VQA-based refinement loop for CAD code generation.
  • CADPrompt benchmark: 200 3D objects (STL + NL prompts + expert-annotated CADQuery code), stratified by mesh complexity, geometric complexity, and compilation difficulty.
  • First quantitative comparison of GPT-4, Gemini 1.5 Pro, and CodeLlama on CAD code generation using Point Cloud distance, Hausdorff distance, IoGT, and compile rate.
  • Demonstration that CADCodeVerify generates model-agnostic feedback (GPT-4 provides visual feedback even for unimodal CodeLlama).

Results

  • GPT-4 few-shot + CADCodeVerify vs. 3D-Premise: Point Cloud distance 0.127 vs. 0.137 (>7% reduction); Hausdorff 0.419 vs. 0.446; compile rate 96.5% vs. 91.0%.
  • GPT-4 zero-shot + CADCodeVerify vs. 3D-Premise: Point Cloud distance 0.132 vs. 0.150; compile rate 94.0% vs. 91.5%.
  • On "Hard" data (compilation difficulty split), 3D-Premise causes a ~20% compile-rate drop at Refine-1; CADCodeVerify achieves ~9% compile-rate increase.
  • Ablation (100 samples, GPT-4 few-shot): removing reference images worsens Point Cloud distance from 0.126 → 0.153; zero-shot QA generation worsens it to 0.141.
  • Human-in-the-Loop comparison (50 samples): human feedback achieves Point Cloud 0.120 / Hausdorff 0.397 vs. CADCodeVerify's 0.137 / 0.445, indicating room for improvement.
  • VQA accuracy: 64.6% at Refine-1, 68.2% at Refine-2 (manually validated subset of 50).
  • Baseline compile rates: GPT-4 96.5% > Gemini 85.0% > CodeLlama 73.5%.

Limitations

  • Point Cloud and Hausdorff distances are noisy spatial metrics that miss logical/structural design flaws (e.g., small gaps that invalidate manufacturability).
  • QA accuracy is only ~65–68%, meaning VLM hallucination during self-assessment degrades feedback quality.
  • Refinement quality is sensitive to the natural-language prompt; multiple valid but geometrically distinct descriptions exist for the same object.
  • CADPrompt covers only 200 objects; dataset annotation relied on a single CAD expert, limiting diversity.
  • The geometric solver baseline (upper bound) requires access to ground-truth STL, which is unavailable in real deployment.

Relevance to Vision-Language Models

This work demonstrates a concrete application of VLMs beyond perception: using their visual question-answering capability to close a code-generation refinement loop in a structured engineering domain. It reveals a practical failure mode — VLM spatial self-assessment accuracy of ~65% — directly relevant to anyone studying VLM hallucination and self-verification. The model-agnostic feedback pattern (a capable VLM acting as a vision-based critic for a weaker unimodal LLM) is a transferable design pattern for agentic VLM systems. CADPrompt also contributes a new out-of-distribution evaluation domain for VLM code and spatial reasoning benchmarks.