Skip to content

SimWorlds: A Multi-Agent System for Dynamic 3D Scene Creation

🕒 Published (v1): 2026-07-02 06:29 UTC · Source: Arxiv · link

Why this paper was selected

Multi-agent orchestration for dynamic scene generation; novel 4D output from LLM agent pipelines

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SimWorlds is a multi-agent framework that converts text prompts into dynamic, editable 4D Blender scenes by running a planner–coder–reviewer loop over a fixed ordered stage sequence, with mechanism correctness enforced by a deterministic orchestrator-side verifier rather than visual inspection alone. It outperforms the only comparable baseline (VIGA) primarily on Mechanism Pass Rate (0.87 vs 0.67), demonstrating that runtime-state inspection catches failures that rendered-image critique cannot.

Problem

Existing LLM-agent pipelines for Blender generate dynamic scenes via render-then-VLM-critique loops, which can only judge visual plausibility. A scene can look correct while relying on the wrong underlying mechanism (e.g., keyframed mesh edits instead of a cloth solver), making it non-editable and physically meaningless as a 4D asset. This failure compounds as scenes grow in complexity—multiple interacting objects, long temporal structure, or multi-solver configurations—because unchecked mechanism errors propagate across construction steps.

Method

SimWorlds decomposes scene generation into a fixed ordered stage sequence \(S = (s_1, \ldots, s_N)\): modeling → UV → texture → deformation setup → motion → camera → light → render. A strategic planner converts the text prompt once into a global scene plan (scene specification, bipartite relation graph, motion phases, per-stage opt-outs). At each stage \(s_k\), a coder extends the live .blend via bpy script; an orchestrator-side verifier runs four families of deterministic rules (structural C1–C6, G1–G3; geometric BVH distance tests V1–V2; soft warnings W1–W5; plan-vs-state checks R1–R18 plus seven motion-timing rules) against Blender's live runtime state; and a reviewer judges perceptual criteria from multi-angle previews and structured runtime readouts. The planner then advances, retries, replans, or aborts—a hard verifier failure cannot advance. A layered scene protocol adds a three-level containment hierarchy (L1 scene → L2 system → L3 logical object) and explicit surface-contact/co-movement declarations on top of Blender's native graph. A blender_docs(query) knowledge base is auto-derived from bpy.types, operator signatures, API guides, and the Blender manual, keyed to the running Blender version.

Key Contributions

  • SimWorlds system: end-to-end text-to-4D Blender pipeline producing editable scenes with geometry, materials, lighting, cameras, animation, and physics all controllable.
  • Staged construction pipeline: fixed-order typed stages with immutable global scene plan, per-stage tactical plans, and bounded retry/replan budgets.
  • Scene protocol + deterministic verifier: declarative L1/L2/L3 hierarchy and bipartite relation graph enforced by orchestrator-side BVH and structural rule checks—mechanism errors caught before any render.
  • Engine-level tool suite: blender_scene_state, multi-granularity preview family, and blender_motion_sheet_preview expose modifier stacks, physics caches, and fcurve channels directly.
  • 4DBuildBench: 50-scene benchmark across five physics solver categories (cloth, fluid, rigid body, particle, soft body) Ă— three difficulty levels, scored on two complementary tracks: deterministic engine-state audit (MPR, SPR) and itemized VLM judge.

Results

  • Overall MPR: SimWorlds 0.87 vs VIGA 0.67 (mechanism correctness; largest gap).
  • Overall SPR: SimWorlds 0.89 vs VIGA 0.70 (spatial relation correctness).
  • Overall VLM score: SimWorlds 0.82 vs VIGA 0.78 (visual fidelity; nearly comparable).
  • Fluid MPR: 0.95 vs 0.56 (largest per-category gap).
  • Rigid body MPR: 0.83 vs 0.61.
  • Static SPR: 0.98 vs 0.42 (static structural layout gap).
  • Gap widens on complex inputs requiring multiple interacting objects, long temporal structure, or nontrivial solver configuration.

Limitations

  • Evaluated against a single baseline (VIGA); other LLM-driven 3D scene systems are not directly runnable on 4DBuildBench due to closed servers, different artifact types, or edit-only modes.
  • 4DBuildBench is small (50 scenes); generalization to broader distributions of prompts is untested.
  • VLM judge uses GPT-5.5, introducing dependency on a proprietary, potentially changing external model.
  • The paper text is truncated; ablation results (§4.3 referenced but not included) are not available for review.
  • Physics bake times and compute cost of the multi-agent loop are not reported.
  • Edit mode (BlenderBench evaluation) results are referenced but not shown in the provided text.

Relevance to Harnesses / Meta-Harnesses

SimWorlds is a concrete instantiation of a stage-gated agent harness: an orchestrator drives a fixed pipeline of typed stages, each with its own context, acceptance criteria, deterministic verifier, and bounded retry budget—a template directly applicable to any multi-step code-generation or construction task. The key harness design insight—separating a deterministic structural verifier (runs every step, catches invariant violations mechanically) from a perceptual reviewer (runs on rendered evidence, catches semantic gaps)—is a general pattern for harnesses where correctness cannot be fully judged from output alone. The planner–coder–reviewer role factorization with per-stage session scoping and checkpoint/rollback mirrors patterns in software build harnesses, and the auto-derived tool-indexed knowledge base (regenerated on version change, no LLM in the build) is a reusable component design for domain-specific agent harnesses.