UnfoldArt: Zero-Shot Recovery of Full Articulated 3D Objects from Text or Image¶

🕒 Published (v1): 2026-06-29 17:44 UTC · Source: Arxiv · link

Why this paper was selected

Laptev + Angela Dai; zero-shot articulated 3D recovery; crucial for embodied AI

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

UnfoldArt is the first debate-driven agentic approach for zero-shot articulated 3D reconstruction from a single text prompt or image. It orchestrates asymmetric LLM agents (global Decomposer vs local Grounder vs Articulator-arbiter) through a two-round structured debate, grounded in a frozen video prior, to resolve articulation parameters and expose occluded interior geometry. A guided video inpainting pass then reconstructs complete interactable URDF assets with high-fidelity hidden interiors.

Problem¶

Recovering articulated 3D objects (part decomposition, joint parameters, and occluded interior structure) from sparse single-state inputs remains unsolved. Supervised methods are limited by small ground-truth datasets (~2k objects from PartNet-Mobility) that lack interior geometry. Existing zero-shot approaches depend on signals unavailable from a single image: multi-view observations across articulation states, a trusted critic backed by a reference asset library, or a video prior fine-tuned on articulation-specific data. None jointly handle articulation estimation and interior geometry recovery from a single static input.

Method¶

Initial geometry: TRELLIS generates a coarse 3D mesh from text/image; Flux Canny-conditioned re-rendering aligns a photorealistic image to the mesh.
Three-agent hierarchy: Decomposer (global — full-image part labels and motion types), Grounder (prompts a segmentation model for 2D masks), Articulator (predicts joint type τ, axis a, pivot p, and range from local crops).
Two-round structured debate:
Round 1 — exploits global-local disagreement. Grounder flags local contradictions (e.g., handle on the proposed pivot edge); Decomposer reconsiders; Articulator consolidates.
Round 2 — grounds the debate in a probe video V_i^probe from a frozen video prior (WAN-VACE). A VLM quality agent gates axis refinement (Rodrigues rotation / translation fitting with optical flow); strict-consensus rule adopts the video-grounded revision or falls back to Round 1.
Interior recovery: Agreed articulation J_i conditions a guided video* V_i^art that drives each part through motion, exposing hidden geometry. A 3D latent inpainting pipeline (RePaint on a voxel grid) labels part vs body voxels from the articulation sweep, resolving only a thin ambiguous boundary shell to produce clean separated meshes.
Output: Interactable URDF with kinematic tree, joint specs, and complete interior geometry.

Key Contributions¶

First debate-driven agentic approach for articulated 3D reconstruction, replacing the standard single-trusted-critic with asymmetric global-local disagreement as the corrective signal.
Two-round debate where the first round exploits agent disagreement and the second grounds articulation in freely generated video (frozen video prior used as evidence, not fine-tuned).
Articulation-conditioned video generation + 3D latent inpainting that recovers occluded interior geometry unrecoverable from a single static view.

Results¶

Evaluated on three datasets of increasing difficulty: PartNet-Mobility (7 classes, 77 objects), Objaverse-Household (12 classes, 53 objects, 8/12 categories novel), and Objaverse-OOD (10 classes, 40 objects including helicopters, robotic arms, humanoids).
Compared against four methods: Singapo and PhysX-Anything (supervised on PartNet-Mobility), FreeArt3D and Articulate-Anything (zero-shot).
On Objaverse-OOD (aggregate table): Ours achieves RS-dgIoU 1.044 vs Singapo 1.049, FreeArt3D 0.768 (needs multi-state inputs), Articulate-Anything 1.387. Supervised Singapo achieves Joint Acc% 37.6, PhysX-Anything 18.8.
Ablates segmentation backends: Gemini (default), SAM3 (stronger on text-recoverable parts), Flux2 (cheaper, joint-prompted as it cannot predict movable parts off-the-shelf).

Limitations¶

Requires a frozen video prior (WAN-VACE) that may not generalize to all articulation types; generated videos are vetted and re-run when part drift or disintegration occurs, adding compute overhead.
Initial 3D geometry depends on TRELLIS, inheriting its reconstruction quality and category coverage constraints.
Debate and video generation pipeline is multi-stage with several LLM calls per part, incurring significant inference cost.
Axis refinement uses optical flow on generated video; the pivot is left to agents since it is more sensitive to track noise.
Evaluated only on revolute and prismatic joints; more complex joint types (screw, universal) not addressed.

Relevance to Vision-Language Models¶

The paper directly leverages VLMs as the core reasoning engine for 3D articulation, demonstrating that VLM priors about object semantics, part structure, and motion families can substitute for supervised articulation data. It also showcases a novel agentic use of VLMs — asymmetric debate with global-local role specialization — that goes beyond single-query prompting and is broadly applicable to other VLM-based 3D reasoning tasks.