ShapeCraft: LLM Agents for Structured, Textured and Interactive 3D Modeling

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ShapeCraft is a multi-agent LLM framework for text-to-3D generation that uses a Graph-based Procedural Shape (GPS) representation to decompose natural language into structured component graphs, which specialized Parser, Coder, and Evaluator agents iteratively refine into structured, textured, and post-modeling-interactive 3D assets. Unlike optimization-based or autoregressive baselines that produce monolithic, difficult-to-edit meshes, ShapeCraft outputs executable Blender API programs per component, enabling direct animation and user editing. It achieves state-of-the-art geometry quality and prompt alignment on the MARVEL benchmark.

Problem

Existing text-to-3D methods—optimization-based (SDS/neural fields) and autoregressive (triangle-sequence generation)—produce unstructured, monolithic meshes with poor semantic part segmentation, topological artifacts, and no support for post-generation editing or animation. Prior LLM-based shape program generation approaches (3D-PREMISE, BlenderLLM) fail because LLMs cannot reliably interpret complex spatial and semantic descriptions without a structured intermediate representation, yielding geometrically inconsistent or non-compiling programs.

Method

ShapeCraft introduces GPS, a flat depth-1 graph G = (V, E, A) where each node vᵢ encodes four attributes: a geometric description nᵍᵢ, a positional description nᵖᵢ, a bounding volume bᵢ ∈ ℝ⁶ (center + size), and an executable Blender Python code snippet pᵢ.

Three specialized agents interact via GPS as shared memory: - Parser hierarchically decomposes the input text into components (e.g., chair→backrest, seat, legs), flattens the hierarchy to leaf nodes, and generates per-node geometric/positional descriptions. - Coder generates bounding volumes and Blender API code snippets per node; runs with M=3 parallel paths at elevated temperature for diversity. - Evaluator (Qwen-VL-Max) renders multi-view images of each component and scores/provides textual feedback to drive iterative refinement.

GPS bootstrapping (N=2 global iterations) refines bounding volume accuracy before shape modeling. Iterative shape modeling then runs T=3 steps per path, with early stopping when score exceeds threshold sτ; the best-scoring path is selected per node. Texturing uses component-aware score distillation sampling (CASD): a learned BRDF texture field ψθ mapping UV coordinates to (diffuse albedo, roughness, metalness) is optimized via SDS loss applied both globally and per-component using component-isolated surface point sets. Post-modeling interaction (animation, editing) is achieved by feeding the stored code snippets directly to an LLM with animation instructions.

Key Contributions

• GPS representation: a structured, flat graph of geometric components with bounding volumes and code attributes that serves as shared agent memory and enables parallel per-component modeling.
• Multi-agent system (Parser + Coder + Evaluator) with iterative multi-path sampling (M=3, T=3) and VLM-driven self-correction.
• Component-aware BRDF painting (CASD) that applies SDS loss at both global and per-component granularity for better text-texture alignment and physically-based rendering.
• Post-modeling interactivity: structured code snippets allow direct LLM-driven animation and user editing without re-optimizing 3D representations.

Results

Evaluated on the MARVEL benchmark subset against MVDream, LLaMA-Mesh, 3D-PREMISE, CADCodeVerify, L3GO, and BlenderLLM:

• IoGT (geometry overlap): 0.471 (ShapeCraft) vs. 0.455 (BlenderLLM), 0.427 (MVDream), 0.385 (3D-PREMISE)
• Hausdorff distance: 0.415 (ShapeCraft) vs. 0.411 (MVDream, best), 0.464 (LLaMA-Mesh); ShapeCraft is close second
• CLIP Score: 27.27 (ShapeCraft, best) vs. 26.99 (BlenderLLM), 26.84 (MVDream)
• VQA Pass Rate: 0.44 (ShapeCraft, best) vs. 0.43 (BlenderLLM), 0.42 (MVDream)
• Compile Rate vs. thinking-mode LLMs: ShapeCraft 100% vs. ChatGPT-o4-mini-high 80%, Deepseek-R1 80%, ChatGPT-o3 60%, Gemini-2.5-Pro 60%
• IoGT ablation: M=3, T=3 (0.471) > M=3, T=1 (0.535 IoGT but worse Hausdorff/CLIP vs full) > M=1, T=1 (0.436); multi-path sampling provides largest geometry gains
• Runtime: 11.68 min (ShapeCraft) vs. 32.10 min (MVDream), 15.64 min (LLaMA-Mesh)

Limitations

• Prompt quality strongly affects output quality; ambiguous, overly brief, or highly creative prompts cause Parser decomposition failures or Evaluator signal degradation.
• Cannot generate complex organic geometry (tails, wings, smooth freeform surfaces) due to Blender API library scope; partially addressed by incorporating native 3D model assets as external components.
• Multi-path + iterative refinement is computationally expensive (21 API calls, 11.68 min per shape); significantly slower than simpler LLM-based baselines.
• GPS representation is flat (depth-1); cannot capture deeply hierarchical spatial relationships naturally.

Relevance to Agentic AI / LLM Agents

ShapeCraft is a concrete demonstration of multi-agent LLM systems with structured shared memory: the GPS graph as a shared, mutable data structure that coordinates heterogeneous agents (linguistic parsing, code generation, visual evaluation) across an iterative refinement loop. The multi-path sampling + VLM-feedback-driven self-correction pattern is directly generalizable to other agentic code-generation tasks where output correctness cannot be verified symbolically but can be assessed visually or empirically. The comparison against chain-of-thought/thinking-mode monolithic LLMs (Table 3) provides empirical evidence that explicit task decomposition via a structured intermediate representation outperforms free-form CoT for spatially complex generation—a key insight for designing agentic pipelines in domains requiring spatial or relational reasoning. The post-modeling interaction capability illustrates how retaining the generative program (rather than a frozen artifact) enables downstream LLM agents to extend or modify outputs, a pattern relevant to tool-augmented and persistent-state agent architectures.