Skip to content

DynaVieW: Schema-Guided World Modeling for Understanding Hierarchical Visual Dynamics

🕒 Published (v1): 2026-07-05 04:44 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML; schema-guided world modeling for hierarchical visual dynamics in multimodal LLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DynaVieW is a schema-guided world model for multimodal LLMs that learns interleaved state-transition sequences to systematically understand hierarchical visual dynamics in videos and multi-image sequences. It jointly predicts structured textual transitions (formatted as hierarchical JSON) and simulates subsequent visual states using a mixture-of-Transformer-experts (MoT) architecture. Pre-training on diverse real-world video data with rich dynamic annotations yields improved consistency, controllability, and instruction-following in downstream tasks.

Problem

Multimodal LLMs fail to systematically model temporal visual dynamics — the multi-level changes (high-level activities, sub-activities, atomic actions, spatial/object/camera transformations) that occur between visual states in sequences. Prior interleaved vision-language pre-training data only weakly aligns visual and textual components, and existing approaches address only narrow dynamic varieties (e.g., motion shifts, robot-view, game-view), leaving general visual dynamic prediction unreliable and uncontrollable.

Method

DynaVieW frames world modeling as learning interleaved sequences of states (video keyframes) and transitions (hierarchical JSON-schema text):

Data Construction: Keyframes are extracted from Ego4D, AgiBotWorld-Alpha, and ShareGPT4Video using Laplacian-variance sharpness selection and CLIP cosine-similarity filtering (threshold 0.925) to ensure quality and diversity. Transitions between adjacent keyframes are annotated by InternVL-78B-Instruct via constrained decoding into a multi-level schema: (a) High-Level Activity → (b) Sub-Activities → (c) Atomic Actions, each linked to seven transformation types (object, object-state, spatial-relation, action, motion, camera, background) and three contribution types (atomic-action, sub-activity, temporal-significance).

Architecture: Built on BAGEL-7B-MoT (initialized from BAGEL-7B-MoT weights), the model uses two Qwen2.5-7B experts sharing multimodal self-attention at every layer: an understanding expert (receiving ViT/SigLIP2 state encodings + text embeddings) and a generation expert (receiving FLUX VAE encodings corrupted with rectified-flow noise). A cross-expert selective attention mask prevents transitions from attending to earlier transitions (blocking copy failures) and limits states to attending only their immediately preceding transition and preceding-state ViT encodings (with 0.3 dropout on earlier states to reduce overfitting).

Training: Two losses are summed with equal weight (1.0 each): (i) a schema token re-weighted cross-entropy for transition prediction, down-weighting static JSON-schema tokens (weight 0.1) and preserving full weight (1.0) on dynamic slot-filling tokens; (ii) a diffusion MSE (velocity prediction) loss for state simulation via the generation expert. Inference samples Gaussian noise into the generation expert and decodes via the VAE.

Key Contributions

  • Hierarchical JSON transition schema capturing 3 activity levels, 7 transformation types, and 3 contribution types for comprehensive visual dynamics annotation.
  • Large-scale, domain-diverse state-transition dataset from Ego4D, AgiBotWorld-Alpha, and ShareGPT4Video with quality-controlled keyframe extraction.
  • MoT architecture with cross-expert selective attention enabling joint transition prediction and visual state simulation over long sequences.
  • Schema token re-weighted CE loss balancing schema-format learning against dynamic slot-value learning.
  • Demonstration that world modeling pre-training on structured dynamic annotations improves downstream visual narrative generation and world simulation.

Results

  • Validation (GPT-4o judge, 900 samples): Transition prediction Reject rate ≤1.89% (Activities), ≤0.53% (Transformations), ≤2.00% (Contributions) — comparable to gold annotations; state simulation Style Consistency 8.54/10 (gold: 8.89/10); simulated state Reject rate <5% with higher Accept and lower None rate than gold (recall-precision tradeoff).
  • VinaBench (Visual Narrative Generation), zero-shot: Higher Average score than Emu2, BAGEL, and Story2Board; particular advantage in cross-scene consistency; strong location consistency attributed to spatially-anchored pre-training dynamics.
  • VinaBench (SFT): Highest Average score among all SFT baselines (ARLDM, MM-Interleaved, StoryGen, BAGEL); balanced improvement across per-scene fidelity and long-range consistency.
  • World simulation (LEGO benchmark): Outperforms baselines on instruction-following under deterministic generation conditions.
  • Controllability: Outputs adapt more flexibly to varying low-level scene descriptions even when the high-level storyline remains fixed.

(Full benchmark tables truncated in the provided text; headline numbers above are as reported.)

Limitations

  • State simulation precision lags gold references by ~2% Reject rate, accepting a recall-over-precision tradeoff.
  • SFT on task-specific data trades cross-scene consistency for per-scene alignment, suggesting tension between expressiveness and coherence.
  • Transition annotations are generated by InternVL-78B-Instruct, which may introduce oracle-model biases and errors propagated into training data.
  • The JSON schema introduces fixed categorical structure; dynamics not fitting the schema's seven transformation types or three activity levels may be poorly captured.
  • The paper text is truncated, so any additional explicitly stated limitations in the conclusion are unavailable.

Relevance to Vision-Language Models

DynaVieW directly targets a core weakness of contemporary multimodal LLMs: inability to model fine-grained temporal dynamics beyond static scene understanding, a gap that limits video and multi-image reasoning. The schema-guided transition representation is a structured bridge between visual perception and language, demonstrating that imposing explicit hierarchical linguistic scaffolding on visual change improves controllability and consistency — a design principle transferable to other VLM training paradigms. The MoT architecture with shared cross-modal attention offers a scalable approach to unifying understanding and generation within a single VLM backbone. This work also contributes a reusable data construction pipeline for rich dynamic annotation, which could seed future VLM pre-training efforts on temporal reasoning.