Skip to content

SceneWeaver: All-in-One 3D Scene Synthesis with an Extensible and Self-Reflective Agent

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SceneWeaver is a reflective agentic framework for 3D indoor scene synthesis that unifies heterogeneous scene generation tools (data-driven generative models, real-to-sim methods, LLM-based generators, VLM refiners) under a standardized interface, coordinated by an MLLM planner executing a reason–act–reflect loop. The agent iteratively selects tools, evaluates its own output on physical and perceptual metrics, and applies physics-based optimization to resolve collisions and boundary violations. It achieves state-of-the-art across common and open-vocabulary room types with zero physical constraint violations.

Problem

No single existing 3D scene synthesis paradigm satisfies all three requirements simultaneously: physical plausibility, visual realism, and open-vocabulary controllability. Rule-based systems lack extensibility; data-driven generative models are constrained to fixed categories and sparse datasets; LLM-based pipelines are "static" (fixed prompts, hard-coded module invocations) and lack adaptive feedback loops, leaving spatial reasoning errors uncorrected. The result is scenes that are physically implausible, lacking in fine-grained detail, or poorly aligned with complex user instructions.

Method

SceneWeaver structures scene synthesis as iterative refinement over up to 10 steps, formalized as a reason–act–reflect paradigm built on the OpenManus/ReAct platform:

  1. Standardized Tool Interface: Existing methods are abstracted as modular tools via tool cards (description, use cases, strengths/weaknesses, supported room types, required inputs), organized into three granularity tiers:
  2. Scene Initializers: data-driven (PhyScene, ATISS), real-to-sim (MetaScene, ACDC), LLM-based (LayoutGPT)
  3. Microscene Implementers: LLM-based layout (GPT) and 2D-guided (Architect, ACDC) for small-object detail

  4. Detail Refiners: VLM rotation correction, rule+LLM relational constraints, LLM-based object removal

  5. Self-Reflective Planner: At each step, an MLLM (GPT-4) generates a reflection comprising physical metrics (collision count, out-of-boundary violations, object count) and perceptual scores (realism, functionality, layout, completeness) plus natural-language justifications. The planner receives this reflection plus a memory window (length=1 to limit hallucination), ranks candidate tools by confidence (dynamically penalizing repeated failures), selects the best tool, and generates tool-specific instructions. Abnormal degradation triggers rollback.

  6. Physics-Aware Executor: Built on Infinigen and Blender; loads the current scene, retrieves 3D meshes from Objaverse/3D-Future/Infinigen pools, enforces relational constraints from refiners, and runs physics-based optimization to resolve collisions and boundary violations at each iteration.

Key Contributions

  • First reflective agentic framework for 3D scene synthesis; introduces the reason–act–reflect paradigm in this domain.
  • Standardized, extensible tool card interface enabling seamless integration of heterogeneous synthesis methods without modifying the agentic core.
  • Feedback-driven self-reflective planner with dynamic tool confidence adjustment and rollback on degradation.
  • Physics-aware executor tightly coupled with the planning loop, enabling quantitative physical-metric feedback.
  • State-of-the-art performance on both pre-defined room categories and open-vocabulary scene generation with zero collision/out-of-boundary violations.

Results

Common room types (bedroom + living room vs. ATISS, DiffuScene, PhyScene, LayoutGPT, Holodeck, I-Design): - SceneWeaver: #OB=0.0, #CN=0.0 (zero physical errors) across both room types - Bedroom: Real.=9.2, Func.=9.8, Lay.=8.4, Comp.=9.4 — highest across all baselines - Living room: Real.=9.1, Func.=9.5, Lay.=8.0, Comp.=8.7 — highest across all baselines

Open-vocabulary generation (8 room types, avg. vs. LayoutGPT, Holodeck, I-Design): - Average #Obj=36.5 (LayoutGPT: 7.3, Holodeck: 22.3, I-Design: 14.3) - Average Real.=8.8, Func.=9.4, Lay.=7.7, Comp.=8.0 — best across all metrics - Zero collisions and out-of-boundary violations on average

Ablation (kitchen scenes): - Removing reflection: Comp. drops 6.3→7.7, Func. 8.3→9.3 (notable semantic degradation) - Removing physics optimization: #CN increases to 2.0, #OB to 0.7 - Multi-step plan (no iterative reflection): Func.=7.7 vs. 9.3, Real.=8.3 vs. 9.0

Human study (20 participants, open-vocabulary setting): - Pairwise preference: 94.3% over I-Design, 91.4% over Holodeck, 87.4% over LayoutGPT - Scene diversity preference: 95.6% / 98.9% / 90.0% over same baselines

Limitations

  • Memory window fixed at length=1 to prevent hallucination, limiting the planner's ability to reason over longer refinement histories.
  • Maximum iteration count capped at 10; complex scenes may require more refinement steps.
  • 2D-guided implementer tools (Architect, ACDC) inherit the spatial reasoning limitations of 2D generation models, leading to inaccurate object rotations.
  • LLM-based tools prone to hallucination without the reflection loop; the system depends on GPT-4 for both planning and metric evaluation, introducing cost and reliability dependency on a closed API.
  • Open-vocabulary generalization still uses fixed room-type lists within initializer tool cards, partially constraining flexibility.

Relevance to Agentic AI / LLM Agents

SceneWeaver is a direct case study in how the reason–act–reflect (ReAct-style) agent architecture scales to a complex, multi-modal, multi-tool domain with quantitative feedback signals — making it a concrete existence proof beyond text-only or code-only agent benchmarks. The paper's central finding — that dynamic tool confidence adjustment and iterative self-reflection outperform single-pass multi-step planning — directly informs general agentic system design. The standardized tool card abstraction is a reusable pattern for building extensible agent toolboxes that can integrate heterogeneous third-party methods without modifying core orchestration logic. For researchers tracking LLM agents in embodied/3D domains, this work bridges agentic orchestration with physics-constrained environment generation, a capability directly relevant to training simulation for embodied agents.