Skip to content

Scenethesis: A Language and Vision Agentic Framework for 3D Scene Generation

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; Scenethesis: agentic framework for spatial-intelligence 3D scene generation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Scenethesis is a training-free, four-stage agentic pipeline that generates physically plausible interactive 3D scenes from text by chaining LLM-based coarse layout planning, vision-foundation-model-guided layout refinement, SDF-based physics-aware pose optimization, and a GPT-5 judge that triggers targeted repair when spatial coherence fails. It outperforms all evaluated baselines on CLIP/BLIP/VQA text-alignment scores and achieves the highest spatial-quality preference in both GPT-5 and human evaluations, while markedly reducing collision and instability rates compared to state-of-the-art methods.

Problem

Learning-based scene generation methods (diffusion/autoregressive) overfit to a handful of indoor datasets (e.g., 3D-FRONT), systematically under-representing small supporting objects and long-tail spatial relations (on-top-of, inside, behind), and failing on out-of-distribution settings such as outdoor scenes. LLM-based layout planners expand semantic diversity but operate in symbolic space rather than metric geometry, producing misoriented furniture, floating objects, and intersecting placements. No prior training-free method enforces mesh-level physics in-loop rather than as a post-hoc correction.

Method

Scenethesis is a four-stage agentic pipeline, all without task-specific training:

  1. LLM Module (coarse planning). Given a text prompt, GPT-5 selects an anchor object, builds a spatial hierarchy, and upsamples the prompt into a coarse layout description with inter-object relations.

  2. Vision Module (layout refinement). A text-to-image model renders the upsampled prompt. Grounded-SAM segments objects; DepthPro lifts segments to monocular depth; back-projection initializes 5-DoF poses (scale, azimuthal rotation, translation). A scene graph encoding parent–child relations is constructed. 3D assets are retrieved from a curated Objaverse subset; environment maps provide wall/background elements.

  3. Physics-Aware Optimization. Given N objects with 5-DoF configurations, dense semantic correspondences from RoMa (robust to partial views) drive pose alignment by minimizing a weighted sum of 2D reprojection and 3D consistency. Separately, SDF-based constraints enforce physical plausibility in-loop:

  4. Collision loss: Surface points with negative SDF values \(d_i < 0\) induce translation correction \(L_\text{translation} = \sum_{v_i \in V^-} \|f(T, |d_i|, u_i) - T\|_2^2\) and scale loss \(L_\text{scale}\) that shrinks objects when multiple collision clusters are detected.

  5. Stability loss: Bottom-surface points are driven to SDF \(\approx 0\) on the parent surface: \(L_\text{stability} = \sum_{v_i \in V^B}(1 - e^{-d_i^2})\). The scene graph hierarchy (anchor → parent → child) is respected; objects are processed in order to establish stable foundations before placing dependent objects.

  6. Judge Module. A GPT-5-based judge evaluates the rendered 3D scene on three normalized metrics—object category accuracy, orientation alignment, and overall spatial coherence. Any metric falling below a threshold triggers re-planning.

Key Contributions

  • Training-free agentic framework unifying LLM planning, vision foundation models (Grounded-SAM, DepthPro, image generators), physics-aware optimization, and a judge-and-repair agent.
  • SDF-based in-loop collision avoidance and stability constraints operating at mesh-surface granularity (not bounding-box approximations), enabling containment/under/on-top-of relations.
  • Generalizes to indoor and outdoor scenes and long-tail spatial relations without retraining.
  • State-of-the-art CLIP (77.17), BLIP (0.8269), and VQA scores against eight baselines; highest GPT-5 and human spatial-quality preference across object diversity, layout coherence, spatial realism, and overall performance.

Results

From Table 1 (text-image alignment vs. baselines DiffuScene, SceneTeller, Holodeck, LayoutGPT, LayoutVLM, IDesign, MIDI, Digital Cousins): - CLIP: 77.17 (next best: IDesign 44.76, Holodeck 46.25) - BLIP: 0.8269 (next best: LayoutGPT 0.8052) - VQA: 51.99 (second, after DiffuScene 48.28 is wrong; Scenethesis 51.99 is actually listed as the highest — SceneTeller 51.99 tied, Holodeck 46.25)

(Note: VQA score for Scenethesis is 51.99, matching SceneTeller and exceeding all others.)

Spatial quality preference (GPT-5 / Human), Scenethesis preferred over each baseline: - vs. DiffuScene: 75%/80% Layout Coherence, 90%/76% Spatial Realism, 80%/80% Overall - vs. Holodeck: 85%/80% Layout Coherence, 81%/86% Spatial Realism, 85%/85% Overall - vs. LayoutGPT: 85%/85% Layout Coherence, 90%/82% Spatial Realism, 90%/71% Overall - vs. IDesign: 70%/74% Layout Coherence, 70%/72% Spatial Realism, 65%/77% Overall

Physical plausibility (Table 2, referenced but not fully shown in the excerpt): Scenethesis markedly reduces collision rate (Col-O, Col-S) and instability rate (Inst-O, Inst-S) compared to all baselines; baselines using bounding-box approximations or post-hoc corrections (Digital Cousins, Holodeck, IDesign) leave overlaps and brittle child-on-parent placements.

Evaluation: 34 prompts (22 indoor, 12 outdoor) across 8 primary categories from DL3DV-10K; 68 rendered images (2 viewpoints per scene).

Limitations

  • Asset diversity is bounded by the curated Objaverse subset; objects absent from the database cannot be placed.
  • Rotation is restricted to azimuthal (upright) adjustments only; tilted or non-upright object poses are not supported.
  • Wall-mounted and background elements are entirely delegated to retrieved environment maps, not individually optimized.
  • Quantitative comparisons with baselines are restricted to the indoor residential subset (most baselines are not designed for outdoor or cross-domain scenes), so the OOD advantage is demonstrated only qualitatively.
  • Depends on proprietary GPT-5 for both the LLM planning module and the judge, introducing API cost and reproducibility constraints.
  • Re-planning is triggered per metric threshold but the number of repair iterations and convergence guarantees are not analyzed in the excerpted text.

Relevance to Agentic AI / LLM Agents

Scenethesis is a canonical example of a modular, tool-augmented agentic loop applied to a structured generation task: an LLM drafts a plan, specialized perception tools (VFMs) ground it in visual reality, a differentiable optimizer enforces domain constraints, and a judge agent closes the loop with targeted repair—a pattern directly applicable to any agent needing to reconcile symbolic LLM outputs with physical or formal constraints. The judge-and-repair mechanism (verify → re-plan on failure) is a concrete instantiation of reflection and self-correction in agents, and the design avoids fine-tuning by composing frozen foundation models as tools, demonstrating training-free generalization through agentic orchestration. For researchers tracking LLM agents, this paper advances the idea that spatial/physical grounding deficits of LLM planners can be remedied architecturally rather than through supervised training, broadening the scope of tasks tractable by agent pipelines.