Skip to content

SceneDecorator: Towards Scene-Oriented Story Generation with Scene Planning and Scene Consistency

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SceneDecorator is a training-free framework for scene-oriented story generation that addresses scene planning and scene consistency—two challenges neglected by prior character-consistency-focused methods. It uses a VLM (Qwen2-VL) to decompose user themes into hierarchical global-to-local scenes, then maintains cross-story scene consistency via a novel Long-Term Scene-Sharing Attention mechanism built on top of SDXL + IP-Adapter.

Problem

Existing story generation methods (ConsiStory, StoryDiffusion, etc.) focus exclusively on character consistency and ignore scene consistency. Two concrete gaps remain unaddressed: (1) scene planning—generating story images from text alone lacks scene-level narrative coherence; (2) scene consistency—maintaining a consistent background/scene across multiple generated stories, especially at scale (>2 stories), is unsolved.

Method

VLM-Guided Scene Planning (global-to-local): 1. A VLM (Qwen2-VL) interprets the user's theme T and generates a global scene description Q. 2. FLUX.1-dev renders a global scene image V from Q. 3. The VLM crops V into M local scenes {V^i} by predicting bounding-box coordinates, then generates N sequential story sub-prompts per local scene.

Long-Term Scene-Sharing Attention (three sub-components, applied during SDXL denoising): - Mask-Guided Scene Injection: Extends IP-Adapter with cross-attention masks derived from subject token activations (A_c). Scene features are blended only into background regions via Z_c^new = A_c·V_c + λ·(1−M)·A'_c·V'_c, preserving subject style diversity. - Scene-Sharing Attention: Extends self-attention so each story branch attends to the other branch's background keys/values (K' = [K, K̃⊙(1−M̃)]), enforcing scene consistency across pairs. - Extrapolable Noise Blending: Generalizes pairwise scene-sharing to N stories by dynamically partitioning latents into N−1 complementary pairs and averaging predicted noises per story, fixing GPU memory at the 2-story level (12.7 GB on RTX 3090) regardless of N.

Key Contributions

  • Formalizes scene-oriented story generation as a distinct problem with two sub-challenges (planning + consistency).
  • VLM-Guided Scene Planning: uses visual understanding of a VLM as a "director" to produce spatially grounded, semantically coherent local scenes from an abstract theme.
  • Mask-Guided Scene Injection: cross-attention masking over IP-Adapter to decouple subject style from background scene injection.
  • Scene-Sharing Attention: background-masked cross-story self-attention for scene coherence.
  • Extrapolable Noise Blending: O(1) memory scaling for long-term (N>2) scene consistency.
  • Training-free and composable with PhotoMaker, ControlNet, LoRA, and manual scene inputs.

Results

  • DreamSim-I (scene alignment, lower is better): SceneDecorator 0.605 vs. ConsiStory 0.723, StoryDiffusion 0.735, CustomDiffusion 0.752.
  • DINO-F (scene consistency, higher is better): SceneDecorator 0.571 vs. ConsiStory 0.475, CustomDiffusion 0.373, StoryDiffusion 0.340.
  • CLIP-T (text alignment): SceneDecorator 0.312, second to ConsiStory (0.320).
  • User study (61 participants, 13 groups, 4 stories/group):
  • Scene alignment: SceneDecorator 76.2% vs. ConsiStory 14.1%, StoryDiffusion 6.3%, CustomDiffusion 3.4%.
  • Text alignment: 56.5% vs. ConsiStory 21.3%.
  • Image quality: 57.5% vs. ConsiStory 24.7%.
  • GPT-4o rationality evaluation of VLM-Guided Scene Planning: Narrative Coherence 90.06%, Theme Adherence 92.57%, Layout Reasonableness 90.29%.
  • Memory: with Extrapolable Noise Blending, 25-story generation uses 12.7 GB (same as 2-story); without it, 20 stories use 23.5 GB and 25 stories OOM on RTX 3090.
  • Evaluated on 2,920 scene-prompt pairs (146 GPT-4o-generated themes × 4 scenes × 5 sub-prompts).

Limitations

  • Scene consistency is enforced pairwise during denoising; very long sequences (large N) may see diminishing coherence despite noise blending.
  • VLM coordinate prediction can produce out-of-boundary crops, requiring a correction heuristic (snap to nearest valid bounding box).
  • CLIP-T (text alignment) is second-best, not first—suggesting a tension between tight scene injection and text fidelity.
  • Relies on off-the-shelf models (Qwen2-VL, FLUX.1-dev, SDXL, IP-Adapter-XL); quality is bounded by those components.
  • No video or temporally consistent generation is evaluated; application scope is static storyboards.

Relevance to Vision-Language Models

SceneDecorator exemplifies VLMs being used not as end-to-end generators but as structured semantic directors: Qwen2-VL supplies spatial reasoning (bounding-box prediction), narrative planning (sub-prompt generation), and perceptual grounding (local scene extraction) in a multi-step pipeline that a pure text-based LLM or diffusion model cannot replicate alone. This positions VLMs as planners in agentic image-generation pipelines—a growing paradigm relevant to embodied agents, creative AI, and multimodal story understanding. The global-to-local scene decomposition is a concrete design pattern for any VLM-in-the-loop generation system where narrative coherence across multiple outputs is required.