Skip to content

LayerCraft: Enhancing Text-to-Image Generation with CoT Reasoning and Layered Object Integration

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LayerCraft is a multi-agent framework for text-to-image generation that uses LLMs as orchestrators, combining CoT-driven layout planning (ChainArchitect) with a dual-LoRA inpainting network (OIN) built on FLUX to produce spatially coherent, multi-object scenes without model fine-tuning. It achieves state-of-the-art on T2I-CompBench and GenEval while supporting batch collage editing and iterative multi-turn refinement.

Problem

Existing T2I systems lack precise spatial composition control for multi-object scenes, require architectural fine-tuning for customization, and fail to maintain object/background consistency across iterative or batch edits. Agent-based predecessors (LayoutGPT, GenArtist) either neglect 3D spatial reasoning or rely on fragmented pipelines with many external models that cause stylistic drift.

Method

LayerCraft comprises three cooperating agents:

  1. LayerCraft Coordinator (GPT-4o, temp=0.1): Central orchestrator. Iteratively enriches user prompts via CoT ("what objects should appear, how should they be arranged"), monitors inter-agent consistency, and supports multi-turn user feedback loops. Skips CoT enrichment if the input is already sufficiently detailed.

  2. ChainArchitect: Performs CoT-style layout planning—identifies objects, generates a background description (passed to FLUX for background image), then constructs a dependency-aware 3D scene graph assigning bounding boxes and inter-object relations (occlusion order, relative positioning, orientation). Uses GPT-4o vision to analyze background viewpoint for placement accuracy. Output is structured JSON via in-context exemplars.

  3. Object Integration Network (OIN): Built on FLUX.1-dev with dual LoRA adapters (rank 4). Phase 1 trains two independent LoRAs—W_bg for inpainting and W_obj for subject-driven generation—using positional/biased embeddings respectively. Phase 2 combines them: the latent sequence is split into [C_T, X, C_bg] and [C_T, X, C_obj], processed in parallel with distinct QKV weight sets (W_both, W_inp, W_obj). The masked region latent from background stream is replaced by the object stream output. No LoRA is active during final image generation; LoRAs only process conditioning inputs.

Training data: IPA300K—300K paired images (object in isolation + in scene) generated via FLUX + Grounding DINO + SAM2 for bounding box extraction, covering 500 object categories × 20 prompts.

Key Contributions

  • Modular multi-agent T2I framework using LLMs as autonomous orchestrators without any task-specific fine-tuning of the LLM components
  • ChainArchitect: CoT-driven 3D-aware layout planning with generation-order and inter-object-relationship modeling
  • OIN: parameter-efficient dual-LoRA adaptation of FLUX enabling reference-guided inpainting while preserving base model quality
  • IPA300K: 300K paired image dataset for image-guided inpainting (to be released on HuggingFace)
  • Support for batch collage editing with consistent identity preservation across multiple images

Results

T2I-CompBench (vs. best prior agent-based method GenArtist): - Numeracy: 0.8643 vs. 0.8482 - Color: 0.7046 vs. 0.6948 - Shape: 0.8147 vs. 0.7709 - Texture: 0.6432 vs. 0.5437 - Spatial: 0.3508 vs. 0.3346 - Non-Spatial: 0.6331 vs. 0.4968

GenEval (overall score): - LayerCraft: 0.84, GPT-4o: 0.84, Janus-Pro 7B: 0.80, SD3.5: 0.71, FLUX.1-Dev: 0.66 - Perfect score (1.0) on Single Object; Two Objects: 0.94 vs. GPT-4o's 0.92; Position: 0.75 (ties GPT-4o)

User study (30 participants, 15 prompts, 1–5 Likert scale): - Prompt consistency: LayerCraft 4.5 vs. GPT-4o 4.4 - Naturalness: LayerCraft 4.5 vs. GPT-4o 4.3 - Overall: LayerCraft 4.2 = GPT-4o 4.2

MagicBrush multi-turn editing: - CLIP-I: 0.9121 vs. GenArtist 0.9071; DINO: 0.8541 vs. 0.8492; CLIP-T: 0.3157 vs. 0.3067

Ablation (on T2I-CompBench, 20% stratified sample): Removing all CoT drops Spatial from 0.3508→0.3013 and Texture from 0.8147→0.7216; removing generation order alone drops Spatial to 0.4210.

Limitations

  • CoT reasoning and multi-agent coordination introduce significant computational overhead, especially for scenes with many interacting objects
  • Efficiency bottleneck lies in agent interaction latency, not per-image generation time
  • OIN trained on only 50K of IPA300K (20K iterations vs. OminiControl's 50K), limiting potential coverage
  • No quantitative runtime/latency comparison against baselines reported

Relevance to Agentic AI / LLM Agents

LayerCraft is a concrete instantiation of the multi-agent orchestration pattern where an LLM coordinator decomposes a creative task into specialized sub-agents—directly relevant to work on agent tool-use, planning, and multi-step task execution. The ChainArchitect exemplifies how CoT reasoning can be embedded into a specialized planning agent that produces structured, machine-readable outputs (JSON scene graphs) consumed downstream by a generative model, a pattern applicable beyond image generation. The coordinator's self-monitoring loop (checking textual/visual consistency and re-delegating on failure) demonstrates a closed-loop agentic architecture with error recovery. Compared to prior agent-based T2I work (GenArtist, LayoutGPT), LayerCraft shows that tight integration between an LLM planning agent and a purpose-trained execution module outperforms loosely coupled pipelines relying on many heterogeneous external tools.