Skip to content

LayoutAgent: A Vision-Language Agent Guided Compositional Diffusion for Spatial Layout Planning

🕒 Published (v1): 2025-09-24 20:41 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LayoutAgent is a multi-stage agentic framework that combines a VLM agent (for segmentation, scene graph construction, and prompt refinement) with compositional diffusion (adapted from robotics) to generate spatially plausible bounding-box layouts for multi-object scenes. A foreground-conditioned image generator then renders the final composite. It outperforms direct VLM prompting and retrieval-based baselines on layout coherence, spatial realism, and aesthetic alignment.

Problem

Text-to-image diffusion models lack explicit spatial reasoning, producing overlapping or mis-scaled objects. Traditional robotic spatial planning methods enforce geometric consistency but cannot capture semantic richness for visual scenes. No existing method bridges high-level semantic reasoning with structured, physically plausible layout generation for multi-object composition.

Method

LayoutAgent decomposes scene generation into four sequential stages, orchestrated by a VLM agent:

  1. Object extraction: GroundingDINO detects objects; SAM or BiRefNet segments and crops them.
  2. Attribute estimation + scene graph construction: The VLM infers physical dimensions \((w_i, l_i, h_i)\) and constructs a scene graph \(G\) encoding pairwise spatial relations (left-of, close-to, in-front-of, etc.); the graph is iteratively refined via a visualization tool.
  3. Compositional diffusion for layout: Each relation type \(r \in R\) is modeled by a separate diffusion model as an energy-based model \(p(P_r | S_r) \propto e^{-E(P_r|S_r)}\). The joint optimum is: $\(P_0 = \arg\min_P \sum_{r \in R} E(P_r | S_r)\)$ solved via annealed ULA. All relation diffusion models are trained jointly with an MSE denoising loss.
  4. Foreground-conditioned image generation: Objects are placed on a canvas per predicted bounding boxes; a text-to-image model (Imagen) renders the final scene with a refined prompt.

Key Contributions

  • LayoutAgent framework: end-to-end pipeline coupling VLM agent planning with compositional diffusion for layout generation.
  • Adaptation of compositional diffusion (from robotics) to 2D bounding-box prediction for visual scene synthesis.
  • Scene graph as an explicit intermediate representation enabling iterative refinement and conflict detection.
  • Ablation confirming that scene graphs and compositional diffusion each independently improve quality, and are complementary.

Results

  • LayoutAgent + Gemini-2.5pro achieves the best scores across all metrics:
  • CLIP: 31.62 vs. 26.52 (SpatialRGPT), 24.32 (best LayoutGPT baseline)
  • BLIP: 74.23 vs. 64.32 (SpatialRGPT)
  • VQA: 0.81 vs. 0.67 (SpatialRGPT)
  • VLM Layout Coherence: 0.82 vs. 0.68 (SpatialRGPT)
  • VLM Spatial Realism: 0.72 vs. 0.64 (SpatialRGPT)
  • VLM Overall: 0.79 vs. 0.69 (SpatialRGPT)
  • On out-of-domain beach scene: LayoutAgent + Gemini-2.5pro VLM Overall = 0.67 vs. 0.58 (Visual Sketchpad), with minimal in-domain/out-of-domain degradation.
  • On 4-object complex scenes: smallest performance drop vs. all baselines; VLM Overall = 0.68.
  • Ablation: VLM alone = 0.56 overall; +scene graph = 0.62; +compositional diffusion = 0.73.
  • Gemini-2.5pro backbone achieves 0.92 success rate and 0.78 IoU on object size estimation vs. GPT-4o at 0.91/0.75.

Limitations

  • Compositional diffusion models trained on only 300 indoor images, limiting coverage of relation types and object distributions.
  • Out-of-domain generalization (e.g., outdoor) is evaluated but the diffusion model is still biased toward indoor spatial statistics.
  • Evaluation is semi-automated (VLM-as-judge), which may inherit VLM biases.
  • The pipeline is multi-stage and sequential; latency and error propagation across stages are not analyzed.
  • Scene graph conflict detection is rule-based and may not catch all semantic inconsistencies.

Relevance to Harnesses / Meta-Harnesses

LayoutAgent is a concrete example of a task-decomposition harness: the VLM agent acts as an orchestrator that sequences tool calls (GroundingDINO, SAM, BiRefNet, visualization) and intermediate reasoning stages (attribute estimation → scene graph → prompt refinement) before handing off to a separate generative module. The iterative scene graph refinement loop—where the agent inspects visualization output and revises the graph—mirrors the self-checking feedback loops central to meta-harness design. The modular, pluggable VLM backbone (GPT-4o vs. Gemini-2.5 variants) further illustrates how harnesses can decouple reasoning orchestration from the underlying model, a key architectural concern for harness builders. The ablation (Table 3) provides rare empirical evidence that structured intermediate representations maintained by the harness (scene graphs) yield larger gains than the generative component alone.