PhysX-3D: Physical-Grounded 3D Asset Generation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PhysX-3D introduces PhysXNet, the first 3D dataset with comprehensive part-level physical annotations (absolute scale, material, affordance, kinematics, function descriptions) covering 26K objects, and PhysXGen, a dual-branch feed-forward image-to-3D generation framework that jointly models structural and physical latent spaces. The annotation pipeline uses GPT-4o with human-in-the-loop refinement to convert geometry-only 3D repositories into physics-annotated assets at scale.

Problem

Existing 3D generative models and datasets focus exclusively on geometry and appearance, ignoring physical properties (mass, material, articulation, affordance) essential for simulation, robotics, and embodied AI. No prior 3D dataset provides part-level annotations spanning all five foundational physical dimensions simultaneously; PartNet-Mobility covers only kinematics for 2.7K objects, and ABO's material/dimension annotations are object-level only.

Method

PhysXNet dataset: A human-in-the-loop pipeline operating in two phases: (1) GPT-4o generates raw annotations (material name, Young's modulus E, Poisson's ratio ν, density ρ, affordance rank 1–10, kinematic type from {no constraints, prismatic, revolute, hinge, rigid, revolute+prismatic}, and three-tier function descriptions) via alpha-composited renders that isolate each part; human experts spot-check and refine. (2) Kinematic parameters (rotation axis, location, range, parent/child part indices) are determined via contact-region extraction from child-parent mesh point clouds, plane fitting, k-means candidate generation, and human selection. A procedurally generated PhysXNet-XL extends coverage to 6M objects.

PhysXGen framework: A two-stage architecture built on top of pretrained TRELLIS (a sparse structured latent 3D generator): - Physical 3D VAE: A physical VAE encoder Ephy maps concatenated physical features Pphy ∈ R^{N×14} (scale, affordance, density, 11-dim kinematic parameters) and CLIP-encoded function description embeddings Psem ∈ R^{N×768×3} into physical latents Pplat ∈ R^{N×8}, parallel to the pretrained structural VAE producing Pslat ∈ R^{N×8}. A residual connection from the physical decoder to the structural decoder (Dep-VAE) enforces cross-branch consistency. - Physical latent diffusion: A dual-branch sparse flow transformer with learnable skip connections from the structural branch into the physical branch generates physical latents conditioned on image features (DINOv2 multi-view average). Both branches are jointly trained via Conditional Flow Matching (CFM): L_diff = L_aes + L_phy.

Key Contributions

• First end-to-end paradigm for physical-grounded 3D asset generation, covering dataset construction through generative modeling.
• PhysXNet: 26K objects with part-level annotations across five physical dimensions; PhysXNet-XL: 6M procedurally generated and annotated objects.
• Scalable human-in-the-loop VLM annotation pipeline converting geometry-only 3D data (PartNet) into physics-first assets using GPT-4o + expert refinement.
• PhysXGen: dual-branch sparse VAE + flow transformer that exploits latent correlations between 3D structure and physical properties without training from scratch.

Results

Evaluated on 1K test samples from PhysXNet against TRELLIS (geometry baseline, no physics) and TRELLIS+PhysPre (post-hoc independent physical property predictor appended to TRELLIS):

• Geometry (PSNR ↑): TRELLIS 24.31 / TRELLIS+PhysPre 24.31 / PhysXGen 24.53
• Geometry (CD ↓ ×10⁻³): TRELLIS 13.2 / TRELLIS+PhysPre 13.2 / PhysXGen 12.7
• Geometry (F-Score ↑ ×10⁻²): TRELLIS 76.9 / TRELLIS+PhysPre 76.9 / PhysXGen 77.3
• Absolute scale (Euclidean dist ↓): TRELLIS+PhysPre 13.21 / PhysXGen 7.24
• Material PSNR (↑): TRELLIS+PhysPre 8.63 / PhysXGen 13.01
• Affordance PSNR (↑): TRELLIS+PhysPre 7.23 / PhysXGen 11.30
• Kinematics COV (↑): TRELLIS+PhysPre 0.24 / PhysXGen 0.33
• Kinematics MMD (↓): TRELLIS+PhysPre 0.12 / PhysXGen 0.08
• Description PSNR (↑): TRELLIS+PhysPre 6.55 / PhysXGen 10.11
• Ablation confirms both Dep-VAE and Dep-Diff components contribute; joint dual-branch training yields the largest gains across all physical metrics.

Limitations

• Struggles with fine-grained property prediction and exhibits mesh artifacts, acknowledged but not analyzed in depth.
• Dataset diversity is limited to synthetic/procedural objects; real-world scan coverage is absent and planned for future work.
• Kinematic type B (prismatic joint) movement range is approximated in 3D coordinate system space rather than true physical movement range due to quantification difficulty.
• PhysXNet scale (26K) is insufficient to train state-of-the-art generative architectures from scratch, necessitating fine-tuning of TRELLIS priors — coupling model performance to an external pretrained system.
• Only five kinematic types are modeled; complex real-world mechanisms (e.g., gear linkages) are outside scope.

Relevance to Vision-Language Models

VLMs are central to PhysXNet's construction: GPT-4o drives the automated annotation phase, processing alpha-composited part renders to infer material properties, affordance ranks, kinematic types, and multi-tier function descriptions for each part at scale. This establishes a new pattern for VLM-powered dataset bootstrapping in physically grounded 3D understanding, where the model's spatial and material reasoning capabilities are stress-tested in a structured, verifiable annotation loop. For researchers tracking VLMs, PhysXGen's use of CLIP embeddings to encode function descriptions into the physical latent space also demonstrates how language-grounded representations can bridge semantic and physical property spaces in generative 3D models. The work positions VLMs as annotation engines for domains (physics simulation, robotics) where ground-truth labels are traditionally expensive to obtain.