Skip to content

VLMaterial: Procedural Material Generation with Large Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLMaterial fine-tunes a large VLM (LLaVA-NeXT with LLaMA 3 8B) to generate Blender procedural materials as Python programs from a single input image. A two-stage data augmentation strategy—LLM-driven evolutionary graph structure crossover followed by parameter perturbation—expands 1,640 curated artist materials to 550K training pairs. The method outperforms all baselines on both in-distribution Blender and out-of-distribution Substance and real-image test sets.

Problem

Inverse procedural material modeling—recovering an editable node-graph material from a single image—requires professional expertise and is underrepresented in VLM pretraining data. Prior work either assumes a fixed graph topology and optimizes only parameters, or trains custom transformers on proprietary (unreleased) Substance 3D datasets, limiting open research. Zero-shot prompting of state-of-the-art VLMs (GPT-4o-mini) fails because procedural material programs are domain-specific and scarce in general pretraining corpora.

Method

Procedural materials in Blender are transpiled into standard Python code, turning inverse material modeling into a code generation task amenable to VLM fine-tuning.

Data collection: 3,663 Blender materials scraped from BlenderKit, Infinigen, and public packs are cleaned (DFS from output node, ≤30 nodes, <2,048 tokens, JPEG size >12 KB heuristic) to yield 1,640 valid materials.

Graph structure augmentation (FunSearch-style evolution): 870 complex materials form a sample pool. GPT-4o-mini performs "genetic crossover" on random pairs, generating novel program structures; valid outputs re-enter the pool. This yields 50.4K structurally distinct programs.

Parameter augmentation: Each program is varied 10× via random parameter perturbation, producing a final dataset of >550K image–program pairs.

Fine-tuning: LLaVA-NeXT (CLIP ViT/L-14 encoder + LLaMA 3 8B decoder + MLP projector) is fine-tuned on these pairs for image-conditioned Python program generation.

Post-optimization: Since Blender nodes lack differentiable counterparts, gradient-free MCMC local search (200 iterations, acceptance probability p_acc=0.05 for worse samples) refines predicted node parameters against a VGG Gram-matrix style loss.

Key Contributions

  • Fine-tuning recipe that adapts a general-purpose VLM to domain-specific procedural code generation via image–program supervision.
  • Two-stage data augmentation combining LLM-driven evolutionary program crossover (graph structure level) with random parameter perturbation, scaling 1.6K → 550K examples.
  • First open-source procedural material dataset for Blender (>550K materials with rendered images).
  • Gradient-free MCMC post-optimization for non-differentiable node graphs.

Results

All comparisons use style loss ↓, SWD ↓, and CLIP ↑; post-optimization excluded unless noted.

  • Blender (in-distribution, n=44): Ours style loss 0.019 vs. GPT-4o-mini 0.041, Cond. MatFormer 0.034, BlenderAlchemy 0.027, Nearest Neighbor 0.026. SWD 1.760 vs. next-best 2.123. CLIP 0.856 vs. next-best 0.778. Program correctness 0.911 vs. GPT-4o-mini 0.294.
  • Substance (out-of-distribution, n=64): Style loss 0.026 vs. best baseline 0.027. SWD 2.283 vs. next-best 2.366. CLIP 0.762 vs. 0.736.
  • Real images (out-of-distribution, n=64): Style loss 0.025 vs. next-best 0.021 (Nearest Neighbor). SWD 2.417 vs. 2.487. CLIP 0.722 vs. 0.700.
  • Ablation: Removing structure augmentation raises Blender style loss from 0.019 to 0.032 and program correctness drops from 0.911 to 0.603; removing parameter augmentation is intermediate (0.022 / 0.897).
  • Post-optimization: Consistently lowers style loss (e.g., Blender 0.019 → 0.015) without degrading CLIP.

Limitations

  • Node expressiveness ceiling: intricate textures requiring >30 nodes cannot be represented, causing failure on fine-grained or highly complex materials.
  • Blender node system is less expressive than Adobe Substance 3D, so the method is outperformed on style loss by Nearest Neighbor on the real-image split—the gap where retrieval of a close match beats generation.
  • No differentiable renderer for Blender means post-optimization relies on slow, stochastic MCMC rather than gradient descent.
  • Evolutionary augmentation uses a commercial LLM (GPT-4o-mini), introducing cost and reproducibility dependence on a proprietary model.
  • Dataset is Blender-specific; cross-platform generalization (e.g., Unreal, Substance) is not demonstrated end-to-end.

Relevance to Vision-Language Models

This paper demonstrates a concrete domain specialization pathway for VLMs: converting a structured, non-linguistic output (a node graph) into a standard programming language to leverage VLM code-generation capabilities, then fine-tuning on synthetic data amplified by a second LLM. It is directly relevant to the growing line of work on VLMs as visual program synthesizers, showing both the failure mode of zero-shot prompting for domain-specific code and an effective mitigation via evolutionary data augmentation. For researchers tracking VLMs, the use of LLM-driven "genetic crossover" for program-level data augmentation is a transferable technique for other low-resource visual programming tasks (CAD, SVG, shader generation). The MCMC post-optimization also addresses a recurring gap when VLM outputs must interface with non-differentiable execution environments.