Video = World + Event Stream¶

🕒 Published (v1): 2026-07-16 14:19 UTC · Source: Arxiv · link

Why this paper was selected

Wan-Streamer (Alibaba); world+event decomposition reframes video VLM architecture

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Wan-Streamer v0.3 proposes a unifying decomposition of video as a world (persistent scene/character/acoustic context) plus an event stream (time-varying changes), turning this into a general-purpose pretraining objective over large-scale natural video. The pretrained model is then specialized to real-time full-duplex audio-visual interaction, where it functions as a vision-language-action (VLA)-like system mapping multimodal user input to interleaved speech and open-vocabulary behavior directives. Critically, this redesign preserves the v0.2 latency operating point: 640×368 at 25 FPS, ~200 ms model-side and ~550 ms total interaction latency.

Problem¶

Prior versions (v0.1/v0.2) trained directly on interaction data for one target application—a talking, visually reactive agent—rather than learning a general competence that transfers across real-time video tasks. This limits pretraining scalability and restricts the agent's expressive behavior to speech and implicit listening cues, with no mechanism for open-vocabulary physical actions.

Method¶

The paper formalizes video as a structured tuple $(W, e_{1:K})$, where: - $W = \{w_j\}_{j \in S(W)}$ is a flexible world record capturing scene, acoustic conditions, and character identities. - Each event $e_k = \langle \tau_k, c_k, d_k \rangle$ is a time-localized record with an interval $\tau_k$, a character association $c_k \in C(W) \cup \{\emptyset\}$, and a free-form description $d_k$.

The model factorizes the joint distribution autoregressively as: $$p_\theta(e_{1:K} \mid W, x_{1:K}) = \prod_{k=1}^{K} p_\theta(e_k \mid W, x_{\leq k}, e_{<k})$$

$W$ is tokenized once as a prefill; the event stream is then generated unit by unit (160 ms per unit) conditioned on causal multimodal history via block-causal attention. Discrete tokens (language + behavior directives) are optimized with next-token prediction; continuous audio/video latents use conditional flow matching. For downstream interaction, the agent's output interleaves spoken words with parenthesized free-form behavior directives (e.g., (picks up the mug and glances toward the window) Sure, one moment.), which jointly condition synchronized audio-video generation without adding a new latency-critical path. The serving topology (Thinker + Ulysses context-parallel performer) is inherited from v0.2.

Key Contributions¶

World + event stream decomposition: A formal reading of any video as a persistent world context $W$ and a time-varying event stream, enabling general-purpose pretraining over unlabeled natural video at scale.
General-purpose pretraining objective: Training to predict how a world streams forward (next event and its audio-visual realization) from large, diverse video corpora rather than interaction-specific data.
VLA-like multimodal understanding: The model maps streaming text, audio, and video input to language-form speech and open-vocabulary behavior actions in an interleaved token stream, enabling grounded, situated agent behavior beyond a fixed action vocabulary.
Latency-neutral expansion: Free-form behavior directives are short language tokens that add negligible cost to the token-causal path, preserving the v0.2 operating point (640×368, 25 FPS, ~200 ms model-side latency, ~550 ms total latency).

Results¶

Latency: ~200 ms model-side signal-to-signal latency (160 ms streaming unit); ~550 ms total interaction latency under a 350 ms bidirectional network budget — identical to v0.2.
Resolution/FPS: 640×368 at 25 FPS, unchanged from v0.2.
Qualitative behavior: In 640×368 generated conversations, behavior directives are realized as coherent facial expressions, posture changes, object interactions, and sound responses, synchronized with speech and consistent with character identity.
No quantitative comparison against external baselines is reported; all latency figures are self-comparisons against v0.1/v0.2.

Limitations¶

No quantitative evaluation against external systems or on standard benchmarks; results are qualitative.
Downstream evaluation covers only the real-time full-duplex audio-visual interaction specialization; transfer to roaming, embodied manipulation, and other stated downstream tasks is left to future work.
Pretraining data curation and scale are not described quantitatively.
World and event annotations used for pretraining are described as "inferred," but the inference pipeline is not detailed.
The paper does not ablate the contribution of world-event pretraining versus interaction fine-tuning.

Relevance to Vision-Language Models¶

Wan-Streamer v0.3 extends the VLM paradigm toward real-time, generative, multimodal interaction by adopting a VLA-like architecture in which vision and language understanding are coupled to action generation (behavior directives + audio-video synthesis) within a single causal Transformer. The world + event stream framing is a principled attempt to turn naturally occurring video into a scalable pretraining signal for VLMs that must reason over persistent context and time-varying change simultaneously—a direct analogue to how static VLMs use image-caption pairs. For researchers tracking VLMs, this work signals a trajectory where vision-language understanding is no longer decoupled from generation and real-time temporal grounding, and where open-vocabulary action specification via natural language replaces fixed action heads.