Skip to content

MAGIC: Transition-Aware Generation of Navigable Multi-Scene Game Worlds with Large Language Models

🕒 Published (v1): 2026-07-13 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MAGIC is a four-stage LLM pipeline that converts a single natural-language prompt into a runnable, multi-scene Unity game project with consistent cross-scene portals and validated navigability. It introduces a formal transition-aware intermediate representation and a flood-fill reachability checker to address failures that single-scene generators cannot resolve by naive repetition. On a new 100-case benchmark, the full pipeline achieves 0.99 precision, 0.95 recall, and 0.96 F1 on end-to-end transition identification.

Problem

Existing LLM/MLLM scene generators (Holodeck, UniGen, WonderVerse, etc.) produce one interior at a time and cannot yield a connected multi-scene world. Three unsolved obstacles remain: (C1) cross-scene consistency — an MLLM prompted independently per scene drops or hallucinates cross-scene portal references; (C2) in-scene navigability — correctly named portals may be physically blocked after furniture placement, and no existing metric detects this; (C3) evaluation gap — all prior text-to-3D metrics measure single-scene visual fidelity and never execute a transition, making broken transition scripts invisible.

Method

MAGIC is a four-stage pipeline:

  1. Planning. An LLM parses the user prompt into per-scene descriptions and a transition-aware automaton \(\mathcal{A} = (S, P, \delta, E)\), where \(S\) is the scene set, \(P\) the portal set, \(E\) transition effects, and \(\delta \subseteq S \times P \times E \times S\) is the (symmetric) transition relation. A verification loop checks portal counts per scene.

  2. Scene Specification. Each scene is expanded with 8–12 objects via layered attribute injection (adapted from Scenethesis's entity extraction). A flood-fill validator operates on a 2D occupancy grid \(G \in \{0,1\}^{r \times c}\) (cell size \(s = 0.05\) units): starting from a portal cell \(g_p\), it expands \(V_{k+1} = V_k \cup \{(i',j') \in \text{Adj}(i,j) \mid (i,j)\in V_k,\, G_{i'j'}=1\}\) and computes \(\text{Connectivity} = |V|/|\{(i,j):G_{ij}=1\}|\). Layouts with connectivity \(< 1\) are regenerated until the maximum iteration limit.

  3. Scene Generation. Meshes are produced via Scenethesis's composition algorithm; LevelLoader transition scripts are attached to portal colliders in Unity, binding each portal to its destination scene and transition effect (FadeInOut or IrisWipe).

  4. Combination. Per-scene Unity projects are merged into one multi-scene project whose scenes reference each other via the emitted scripts.

A transition-focused evaluation agent (using an MLLM judge for portal appearance) executes each transition in play, navigates from spawn to each portal, and reports Precision, Recall, F1, Approach Rate, and Portal Match Rate against ground-truth scene plans.

Key Contributions

  • End-to-end prompt-to-Unity pipeline for navigable multi-scene game worlds using a shared transition-aware IR enforcing cross-scene consistency.
  • A 100-case multi-scene benchmark derived from MIT Indoor 67 and MMIS, covering looping/branching transition patterns and mixed portal types — the first benchmark with ground-truth cross-scene portals and reachability labels.
  • A transition-focused MLLM evaluation agent that executes transitions at runtime, complementing static single-scene fidelity metrics.
  • Stage-by-stage and end-to-end ablation establishing where gains originate in the pipeline.

Results

  • End-to-end transition identification: 0.99 precision, 0.95 recall, 0.96 F1 (vs. LLM baseline and Holodeck).
  • MAGIC is the only evaluated system that produces an executable Unity project for all 100 benchmark cases.
  • Stage-by-stage: MAGIC recovers more ground-truth portals and yields markedly more navigable layouts than both the LLM baseline (GPT-4.1) and Holodeck at the scene specification stage.
  • Median generation time: 33.35 min/scene (mean 46.36 min), right-skewed.
  • Backbone LLM: GPT-4.1-mini-2025-04-14 (MAGIC) vs. GPT-4.1 (LLM baseline).

Limitations

  • Generation is slow (median ~33 min/scene); scaling to large scene counts is impractical.
  • Transition effects are limited to two options (FadeInOut, IrisWipe); richer cinematic transitions are not supported.
  • The flood-fill validator operates on a 2D projection and cannot detect 3D-specific blockages (e.g., vertical occlusion).
  • Object placement relies on an LLM with "weak 3D spatial perception," requiring iterative correction loops that may not converge within the iteration budget.
  • The benchmark is procedurally generated from two indoor-scene datasets; outdoor or fantastical game environments are not covered.
  • Paper was truncated before full quantitative stage-by-stage breakdowns were visible.

Relevance to Vision-Language Models

The transition-focused evaluation agent is the primary VLM touchpoint: it deploys an MLLM as a judge to assess portal visual appearance during runtime play, integrating visual grounding with spatial navigation — a concrete use of VLMs for environment-level evaluation rather than single-image assessment. The paper also situates itself within the broader text-to-3D landscape where MLLMs (e.g., Holodeck) have become the dominant paradigm for single-scene synthesis, making MAGIC's multi-scene extension a direct forward step for VLM-driven procedural content generation. For VLM researchers, the introduced benchmark and evaluation metrics offer a new testbed where a model must reason about cross-scene spatial consistency, not just within-image semantics. The system architecture — using MLLMs for layout generation while delegating geometric correctness to a symbolic validator — illustrates a hybrid approach that could inform how VLMs are embedded in structured spatial pipelines.