ShapeCraft: LLM Agents for Structured, Textured and Interactive 3D Modeling¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

ShapeCraft is a multi-agent LLM framework for text-to-3D generation that represents 3D assets as executable shape programs via a Graph-based Procedural Shape (GPS) representation. Three specialized agents—Parser, Coder, and Evaluator—collaborate through a shared GPS data structure to iteratively produce structured, textured, and editable meshes in Blender. It outperforms both optimization-based and LLM-based baselines on geometry quality and prompt alignment.

Problem¶

Existing text-to-3D methods produce either unstructured meshes with topological artifacts (optimization/diffusion-based) or monolithic sequences with poor editability (autoregressive). LLM-based shape program generators fail on complex spatial/semantic descriptions because LLMs lack fine-grained spatial grounding, yielding geometrically inaccurate programs.

Method¶

ShapeCraft decomposes input text into a flat GPS graph $\mathcal{G} = (V, E, A)$ where each node $v_i$ holds four attributes: geometric description $n_i^g$, positional description $n_i^p$, bounding volume $b_i \in \mathbb{R}^6$, and a Blender API code snippet $p_i$.

Parser hierarchically parses input text, flattens the hierarchy to a depth-1 graph, and generates per-node descriptions.
Coder generates bounding volumes and code snippets; runs $M=3$ parallel sampling paths per node at high temperature to explore design diversity.
Evaluator (VLM) renders partial geometry via the procedural executor $\Omega$ (Blender), scores each candidate ($s_{i,m}^t$), and returns textual feedback $f_{i,m}^t$.

Iterative refinement runs for $T=3$ steps per node with early stopping if $s_{i,m}^t \geq s_\tau$; the best-scoring path is selected: $v_i^* = \arg\max_m s_{i,m}^{\text{best}}$.

A representation bootstrapping loop ($N=2$ iterations) refines bounding volumes before code generation begins.

Texture is applied via a Component-aware Score Distillation (CASD) scheme: a learnable UV texture field $\psi_\theta: \mathbb{R}^2 \to \mathbb{R}^5$ (BRDF parameters $k_d, k_r, k_m$) is optimized by combining a global SDS loss with per-component SDS losses: $$\mathcal{L}_{\text{CASD}} = \mathcal{L}_{\text{SDS}}(\mathcal{L}(\psi_\theta(p), \omega), x) + \sum_{i=1}^{M} \mathcal{L}_{\text{SDS}}(\mathcal{L}(\psi_\theta(p_{v_i}), \omega), n_i)$$

Key Contributions¶

GPS representation: a flat, component-structured graph that serves as shared memory across all agents, enabling parallel per-component modeling.
Multi-path sampling with iterative VLM feedback per node, increasing robustness against single-path LLM failure.
Representation bootstrapping: a bounding-volume refinement loop that corrects spatial errors before code generation.
Component-aware BRDF-based texture painting that leverages GPS decomposition for fine-grained text-texture alignment.
Post-modeling interactivity: the structured representation directly enables shape editing and animation.

Results¶

On the MARVEL subset (compared against 3D-PREMISE, CADCodeVerify, BlenderLLM, LLaMA-Mesh, MVDream; all LLM-based methods use the same backbone Qwen3-235B-A22B):

IoGT (intersection over ground truth): 0.471 vs. next best BlenderLLM 0.455
Hausdorff Distance: 0.415 vs. best MVDream 0.411 (close second)
CLIP Score: 27.27 vs. next best BlenderLLM 26.99
VQA Pass Rate: 0.44 vs. next best MVDream 0.42
Runtime: 11.68 min vs. MVDream 32.10 min; faster than LLaMA-Mesh (15.64 min)
API call count: 21 (higher than simpler LLM baselines with 6–9 calls, but N/A for neural methods)

Limitations¶

Runtime (11.68 min) is substantially longer than simpler LLM-based baselines (3–5 min) due to multi-path sampling and iterative VLM evaluation.
Higher API call count (21) increases inference cost relative to single-pass methods.
GPS uses a depth-1 flat graph, which may struggle with deeply hierarchical objects requiring nested geometric dependencies.
Relies on Blender Python API for execution; shapes outside the API's primitive vocabulary require complex, LLM-generated code that remains error-prone.
Texture quality ultimately depends on SDS optimization, which can still produce blobby or over-smooth appearance in high-frequency detail regions.
Evaluation is on a single benchmark subset (MARVEL); generalization to broader shape categories is qualitative only.

Relevance to Harnesses / Meta-Harnesses¶

ShapeCraft is a concrete instantiation of a domain-specific multi-agent harness: a structured orchestration layer that routes sub-tasks across specialized agents (Parser, Coder, Evaluator) through a shared intermediate representation (GPS graph). The iterative bootstrapping loop and multi-path sampling with early stopping are harness-level control-flow patterns—comparable to retry/validation loops and ensemble sampling in LLM meta-harnesses. The GPS graph functioning as a "shared memory" that coordinates agent handoffs is a design pattern directly relevant to state-management strategies in agentic harnesses. For researchers tracking harness design, this paper demonstrates how a structured intermediate representation can replace ad-hoc prompt chaining, improving both correctness and debuggability.