Skip to content

3D-DefectBench: A Controlled Factorial Study of Vision-Language Model Evaluation Pipelines for Fine-Grained 3D Generation Defects

🕒 Published (v1): 2026-07-12 16:41 UTC · Source: Arxiv · link

Why this paper was selected

Factorial study of VLM eval pipelines for 3D generation; rigorous methodology for automated judge reliability

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

3D-DefectBench treats VLM-based evaluation of 3D generation as a configurable measurement pipeline problem rather than a model-selection problem, and uses a balanced factorial design across 84 inference configurations (VLM × camera protocol × visual input × prompt schema) to disentangle pipeline effects. The dominant source of agreement variance is model choice, but pipeline factors also materially interact with the model. A compact six-view RGB protocol matches denser configurations and depth/normal augmentation, while the best of 12 tested VLMs still trails trained human labelers.

Problem

Existing VLM-based 3D-generation evaluators fix rendering, view layout, and prompt design as implementation details and vary only the judge model, confounding model capability with pipeline choices. Moreover, holistic or pairwise preference scores provide limited actionable signal for generator development, and calibration of automated judges against human reference labels is rarely examined across multiple annotation quality regimes.

Method

Benchmark construction. Assets are 1,049 silver-labeled (≥3 vendor-annotator votes per asset; silver ground truth = per-defect majority vote) and 129 expert-labeled (two 3D-artist authors of the rubric) generated textured GLB meshes from two variants of Roblox's Cube text-to-3D system. Each asset receives a nine-dimensional binary defect vector: five geometry defects (form/surface quality, fused/incomplete parts, pose/placement, missing parts, extra geometry) and four texture defects (noise/blur, misplaced/overlapping texture, baked lighting/shadow, visual-textual mismatch).

Factorial design. Four pipeline factors are fully crossed: - Camera-view protocol (3 levels): 6-view oblique turntable, 14-view equatorial, 14-view multi-ring - Visual input (7 levels): RGB alone or paired with depth/normal/geometry channels; depth-only; normal-only; geometry-only - Prompt schema (4 levels): Compact Binary → Definition-Guided Binary → Rubric-Guided Binary → Rubric-Guided Checklist (increasing rubric detail) - VLM (5 screening models): Gemini 3.1 Flash-Lite, GPT-5 Mini, Qwen 3.5 397B-A17B, Mistral Small 3.1 24B, and one additional model

This yields 84 inference configurations. A defect-level logistic factor model estimates main effects and pairwise interactions.

Staged evaluation. (1) Screen all 84 configs on 1,049 silver assets using the 5 cost-effective VLMs (692,340 API calls). (2) Validate design conclusions on a broader 12-VLM frontier set. (3) Fix the near-optimal configuration c004 (6-view oblique RGB + Rubric-Guided Checklist prompt) for comprehensive comparison against silver holdout (549 assets), expert labels (129 assets), and a trained human leave-one-out benchmark.

All views are rendered offscreen at \(512 \times 512\) per view and packed into a single grid image; geometry channels use flat ambient lighting only.

Key Contributions

  • Nine-category binary defect taxonomy spanning geometry and texture, with silver (majority vote) and expert (two 3D artists) reference regimes on 1,049 + 129 assets
  • Balanced factorial framework over 84 inference designs; defect-level logistic factor model to estimate main effects and model × pipeline interactions
  • Cost-aware staged methodology that decouples optimal pipeline selection (screening phase) from VLM comparison (validation/fixed-pipeline phases)
  • Calibration analysis comparing 12 VLMs against both reference regimes, a trained human labeler benchmark, and downstream generator-comparison stability
  • Public release of labels, prompts, predictions, and Croissant metadata on HuggingFace

Results

  • Model choice is the dominant source of defect-agreement variation across the 84 designs; remaining pipeline factors also have measurable main effects and model-interaction effects
  • 6-view oblique RGB (c004) performs comparably to 14-view protocols and depth/normal-augmented inputs, making it the preferred cost-effective default
  • Best of 12 VLMs trails trained human labelers under matched protocols (no numerical gap stated in the excerpted text)
  • Texture inter-annotator agreement is markedly lower than geometry: silver Fleiss' \(\kappa\) ranges 0.117–0.254 for texture vs. 0.085–0.309 for geometry; expert Cohen's \(\kappa\) ranges 0.277–0.359 for texture vs. 0.380–0.566 for geometry
  • Texture VLM agreement drops sharply when moving from expert-agreement labels to noisier silver labels
  • Total scored cells across all phases: 3,283,178 (screening: 3,209,954; silver holdout: 59,292; expert eval: 13,932)

Limitations

  • Assets are from only two variants of a single proprietary system (Roblox Cube); defect distribution was resampled to support judge-pipeline evaluation and is not production-representative
  • Background color and lighting conditions were held fixed; sensitivity to those rendering choices is unexamined
  • Geometry/texture label-noise ceiling constrains achievable VLM agreement in a way not fully separable from model capability
  • Extreme class imbalance in rare defects (e.g., pose/placement prevalence = 0.012) depresses Fleiss' \(\kappa\) even under good agreement; authors note this and provide Randolph free-marginal \(\kappa\) as a robustness check
  • Conclusions are scoped to text-to-3D with short, single-concept object prompts; generalization to compositional or scene-level prompts is untested

Relevance to Vision-Language Models

This paper directly addresses the reliability of VLMs deployed as automated judges—a fast-growing evaluation paradigm—by showing that pipeline choices (rendering, view count, prompt verbosity) interact with model capability and can shift which VLM appears best. The finding that a compact six-view RGB input matches depth/normal-augmented inputs challenges the assumption that richer visual channels always help VLMs, and the calibration gap between VLMs and trained humans on fine-grained texture defects points to a specific perceptual weakness for current models. The factorial methodology and multi-regime calibration approach are broadly transferable to any domain where VLMs are used as judges over constructed (non-natural) visual inputs, including image editing, video generation, and embodied-AI evaluation.