Skip to content

Streaming Multi-Agent Autoregressive Diffusion Model with World State Registers

🕒 Published (v1): 2026-07-23 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WorldWeaver (W²) is a streaming multi-agent video diffusion model that augments autoregressive rollout with explicit world state registers (WSR) — learnable tokens shared across agents that are persistently updated each generation step. By grounding these registers with agent status, bird's-eye-view maps, and scene text supervision, the model achieves significantly better logical consistency and cross-agent coherence than approaches that rely solely on local frame history.

Problem

Chunk-based autoregressive video diffusion pipelines condition only on a local sliding window of past pixel-space frames, forcing the model to re-infer world state at every step. In multi-agent, multi-view settings this is especially problematic: each agent sees only a partial 2D projection of a shared 3D world, so frame-history conditioning cannot maintain a coherent shared state between agents or account for changes in unobserved regions.

Method

WorldWeaver interleaves world state register tokens \(r_i \in \mathbb{R}^{K \times d}\) with frame tokens in a causal transformer as \([x_0, r_0, x_1, r_1, \ldots]\). Registers attend to the local frame window \([x_{i-W+1}, \ldots, x_i]\) and the previous register \(r_{i-1}\), and are committed after each denoised chunk to condition the next one: $\(p_\theta(x_{i+1} \mid x_{i-W+1}, \ldots, x_i, a_{i+1}, r_i), \quad r_i = G_\theta(r_{i-1}, x_{i-W+1}, \ldots, x_i, a_i)\)$

Training follows a three-stage curriculum: 1. Stage 1 (Bidirectional): Fine-tune a multi-player teacher with full bidirectional cross-agent attention and flow-matching loss \(\mathcal{L}_\text{flow}\). 2. Stage 2 (Causal + WSR): Convert to causal student with block-level causal masking and add register supervision \(\mathcal{L}_{S2} = \mathcal{L}_\text{flow} + \lambda_\text{state}\mathcal{L}_\text{reg}\), where \(\mathcal{L}_\text{reg}\) supervises registers via three auxiliary decoders: agent status (\(\ell_2\) on position/velocity/orientation), bird's-eye-view (cosine distance against frozen DINOv2 features), and scene text (cross-entropy on Qwen2.5-VL-72B annotations). 3. Stage 3 (Self-Forcing): Roll out the student on its own predictions, applying DMD-style distribution matching from the Stage 1 teacher, plus retained register supervision \(\mathcal{L}_{S3} = \mathcal{L}_\text{sf} + \lambda_\text{state}\mathcal{L}_\text{reg}\).

A Mixture-of-Transformers (MoT) design assigns separate weight sets to register and frame token pathways while maintaining joint attention, reducing conflicts between state updating and frame generation objectives. Auxiliary decoder heads are discarded at inference, adding no rollout cost.

Key Contributions

  • World state registers: Persistent, dynamically-updatable learnable token groups shared across agents and rollout steps, carrying global scene state beyond the local context window.
  • Multi-signal register grounding: Supervision spanning agent status vectors, DINOv2-supervised bird's-eye-view maps, and VLM-generated scene text; enables semi-supervised training where a small labeled subset anchors register semantics.
  • MoT architecture for video diffusion: Separate transformer weights for register vs. frame pathways, shown to reduce objective conflict and stabilize long-horizon self-forcing.
  • Three-stage curriculum: Bidirectional teacher → causal student with register supervision → self-forcing rollout with frame and state drift exposure.

Results

Evaluated on two-agent Minecraft video generation (Solaris test split, \(K=5\) categories) using VLM accuracy, FID, and an auxiliary WorldScore \(= \frac{\bar{\text{VLM}}}{\bar{\text{FID}}} \times 100\).

  • WorldScore: W² achieves 105.1 vs. Solaris 81.0 and Frame-concat 49.1 (best by large margin).
  • Grounding VLM accuracy: 93.8% vs. Solaris 81.3% and Frame-concat 53.1%.
  • Building VLM accuracy: 28.1% vs. Solaris 9.4% and Frame-concat 0.0%.
  • Consistency VLM accuracy: 76.6% vs. Solaris 57.8% and Frame-concat 49.5%.
  • Movement FID: 34.0 vs. Solaris 43.3 and Frame-concat 68.9 (lower is better).
  • Baselines: (1) Frame-concat (channel-wise concatenation, à la Multiverse), (2) Solaris retrained on identical data.

Limitations

  • Experiments confined to two-agent Minecraft; generalization to more agents, different environments, or real-world video is unverified.
  • Scene text annotations depend on Qwen2.5-VL-72B-Instruct, introducing potential annotation noise and a strong external model dependency.
  • WorldScore is explicitly noted as auxiliary and unbounded — sensitive to FID scale, not suitable as a primary metric.
  • The paper text is truncated before ablation analysis (Sections 4.3–4.5), so quantitative ablations of individual supervision signals, MoT architecture, and semi-supervised training are not reported here.
  • Register drift in Stage 3 required increased per-head supervision weights to stabilize; the DMD loss alone was insufficient, suggesting fragility in long-horizon register coherence.

Relevance to Agentic AI / LLM Agents

WorldWeaver directly addresses a core challenge in agentic systems operating in shared environments: how to maintain a consistent, persistent representation of world state that is observable to multiple agents with partial views. The world state register mechanism is architecturally analogous to a shared external memory or workspace for agents — a design principle relevant to multi-agent LLM architectures where agents must coordinate around shared context. The multi-signal grounding approach (structured state, spatial maps, language) is directly transferable to LLM agent worlds where structured + natural-language state representations are common. The self-forcing curriculum for reducing rollout drift under register conditioning is a general technique for training any agent that must act on its own past outputs over long horizons.