Skip to content

ShapeLLM-Omni: A Native Multimodal LLM for 3D Generation and Understanding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ShapeLLM-Omni extends Qwen-2.5-VL-7B-Instruct to natively handle 3D mesh generation, understanding, and editing alongside text and images within a single autoregressive model. It encodes 3D shapes into 1024 discrete tokens via a 3D VQVAE and fine-tunes on the newly constructed 3D-Alpaca dataset (3.46B tokens, 2.56M samples). This is the first unified model supporting text-to-3D, image-to-3D, 3D captioning, and 3D editing in a single interactive framework.

Problem

Existing multimodal LLMs (including GPT-4o) are restricted to text and images, leaving 3D content understanding and generation to task-specific systems. Prior 3D-aware LLMs either treat mesh OBJ files as raw text (LLaMA-Mesh), rely on point clouds that lack fine geometric detail and are hard to acquire, or cannot jointly generate and understand 3D content in an interactive, instruction-following paradigm. No single model unifies all four 3D tasks—generation (text/image-conditioned), understanding (captioning), and editing—under a next-token prediction framework.

Method

3D VQVAE tokenization. 3D meshes are voxelized at 64³ resolution. A 3D U-Net encoder compresses this to a 16³ latent grid (4096 tokens with 8 channels). Neighboring token groups of 4 are channel-concatenated, yielding 1024 tokens with 32 channels. An 8192-entry codebook quantizes these into 1024 discrete 3D tokens per object. A Rectified Flow model (from Trellis) reconstructs high-quality meshes from the coarse voxel predictions.

Model architecture. Qwen-2.5-VL-7B-Instruct is extended with the 3D VQVAE codebook (8192 new entries). Text and 3D are represented as discrete token sequences (early fusion); images remain as continuous features since images only appear in understanding inputs, not generation targets. All modalities are processed in a single transformer via next-token prediction.

3D-Alpaca dataset. 712k 3D assets (from Objaverse-XL, ABO, ShapeNet, Trellis) are processed into four task types: text-to-3D (0.77B tokens), image-to-3D (1.01B tokens), 3D-to-caption (0.77B tokens), and 3D-editing (0.91B tokens, 420k pairs). Editing pairs are constructed via a six-step pipeline: classify assets into 100 categories, generate editing prompts with GPT-4o, render and edit images with GPT-4o, then reconstruct edited images back to 3D via Trellis. UltraChat is mixed in for general conversation retention.

Training. 3D VQVAE: two-stage (codebook-only first, then joint fine-tuning), 1000 steps each, 48× H100. ShapeLLM-Omni: 15 epochs on 48× H100, lr 5×10⁻⁵→5×10⁻⁶, visual encoder frozen.

Key Contributions

  • First unified autoregressive LLM supporting text-to-3D, image-to-3D, 3D captioning, and 3D editing in a single interactive framework.
  • 3D VQVAE that compresses 64³ voxel grids into 1024 discrete tokens with an 8192-entry codebook.
  • 3D-Alpaca: a 3.46B-token, 2.56M-sample training corpus spanning four 3D tasks, released for the community.
  • Demonstration that a 7B unified model can approach task-specific specialist performance while retaining language reasoning.

Results

  • 3D VQVAE reconstruction (1000 test samples): IoU=0.917, F1=0.945, Precision=0.955, Chamfer Distance=0.021.
  • Text-to-3D (Toys4K): CLIP=26.7, FD=25.9, KD=0.25×10⁻² — below Trellis (CLIP=30.8, FD=18.3, KD=0.19); above SAR3D (CLIP=23.9, FD=27.2, KD=0.28).
  • Image-to-3D (Toys4K): CLIP=84.5, FD=12.2, KD=0.09×10⁻² — below Trellis (CLIP=85.0, FD=8.31, KD=0.07); above SAR3D (CLIP=84.7, FD=20.6, KD=0.17) and CRM (FD=14.7, KD=0.12).
  • 3D captioning (Objaverse, PointLLM benchmark): B-1=18.51, ROUGE-L=21.37, METEOR=19.89, S-BERT=49.34, S-CSE=50.72 — exceeds all baselines (ShapeLLM, 3D-LLM, GPT4Point, LLaVA-13B, InstructBLIP-13B) except PointLLM-13B* (B-1=17.09, R-L=20.99, S-BERT=50.15) which is a single-task specialist.
  • Language retention (MMLU/PIQA/GSM8K/SIQA): 63.9/78.6/55.1/41.0 vs. Qwen2.5-vl-7B base at 66.9/81.0/42.9/40.7 — comparable, with some metric shifts due to domain shift from 3D fine-tuning.
  • Task-specific fine-tuned ablation (text-to-3D only): CLIP=30.1, FD=18.9, KD=0.21 — nearly matches Trellis, confirming multi-task trade-off is the primary gap.

Limitations

  • Only 70k 3D-editing pairs; editing quality falls well short of ChatGPT-4o-level.
  • 7B parameter cap due to compute constraints limits overall capacity.
  • Autoregressive discrete token generation is inherently weaker than diffusion/flow-based architectures (Trellis) for visual generation fidelity; the gap is acknowledged as an open problem.
  • Voxel representation at 64³ is coarse; fine geometric detail depends on the downstream Trellis Rectified Flow reconstructor.
  • Text-to-3D lags Trellis when all tasks are trained jointly; specializing the model closes the gap but destroys general capabilities.

Relevance to Vision-Language Models

ShapeLLM-Omni is a direct 3D analog of how GPT-4o natively integrates image generation into a language model, demonstrating that the VLM tokenization paradigm (discrete codebooks, early fusion, next-token prediction) extends naturally to a third spatial modality. For VLM researchers, it highlights the central design tension between unified multimodal generation and modality-specialized quality. The 3D-Alpaca dataset and VQVAE tokenizer are practical infrastructure contributions for training 3D-aware VLMs, and the results establish a baseline for how much task-specific performance is sacrificed by joint training. The work sits at the frontier of expanding VLMs from 2D perception to full 3D spatial intelligence, with implications for robotics, embodied AI, and spatial reasoning.