Skip to content

Infinite Worlds with Versatile Interactions

🕒 Published (v1): 2026-07-08 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LingBot-World 2.0 (LingBot-World-Infinity) is a causal autoregressive video world model that achieves unbounded, drift-free interactive generation at 720p/60 fps via a two-stage distillation pipeline. Its key agentic contribution is a Director-Pilot Co-Simulation Framework in which a VLM handles causal semantic reasoning while a Diffusion Transformer handles physical video synthesis. The system supports rich interaction (combat, archery, spell-casting, weather events) and multi-player sessions, verified stable over 1+ hour of continuous generation.

Problem

Existing interactive video world models suffer from two fundamental limitations: (1) long-horizon drift, where autoregressive error accumulation degrades visual quality within minutes, and (2) latency-fidelity tradeoffs, where real-time interactivity requires sacrificing resolution or semantic control. Neither limitation has been resolved simultaneously in a general-domain, open-source system.

Method

Causal pretraining formalizes world simulation as: $\(p_\theta(x_{1:T} \mid a_{1:T}) = \prod_t p_\theta(x_t \mid x_{<t}, a_{\leq t})\)$ trained via a conditional flow-matching objective. The key attention mechanism is MoBA (Mixture of Bidirectional and Autoregressive) Mask: teacher-forcing autoregressive attention is augmented with a bidirectional block over all frames, acting as a regularizer that prevents the model from over-relying on context and degrading under long rollouts.

Two-stage post-training distillation compresses the multi-step base model into a few-step student: (i) consistency distillation enforces trajectory invariance on the teacher PF-ODE, and (ii) distribution matching distillation (DMD) minimizes \(D_\text{KL}(p_{\theta,t} \| p_{\text{data},t})\) over long self-rollout trajectories (not teacher-forced states), specifically targeting rollout drift.

The agentic harness uses a VLM as Director (scene analysis → event proposals) and the video DiT as Pilot (grounding proposals into spatio-temporal frames). Two interaction modes: Mode A (direct semantic, holistic) and Mode B (SAM-tracked object-centric). A hierarchical multi-track annotation pipeline provides chunk-wise, temporally localized captions across separate event tracks (motion, interaction, scene dynamics) to eliminate train–inference mismatch.

Key Contributions

  • Causal world model with MoBA attention that resists long-horizon drift; serves as a distillation teacher.
  • Real-time distilled model (14B → few-step student) sustaining 720p/60 fps generation verified stable over 1+ continuous hour.
  • Extended action vocabulary: combat, archery, spell-casting, ranged fire, and on-the-fly environmental changes (weather summoning).
  • Director-Pilot agentic harness: VLM provides causal semantic reasoning; video generator executes physical simulation; together they produce self-sustaining, goal-directed world behavior.
  • Multi-player interface enabling simultaneous co-immersion in the same simulated world.
  • 14B + 1.3B model pair: the lightweight variant targets single-GPU deployment.
  • Hierarchical data engine with multi-track event-level chunk annotations for temporal controllability.

Results

The provided text contains only a qualitative comparison table (Table 1) against M-G 3.0, D-W, LingBot-World, HappyOyster, and Genie 3:

  • Generation duration: All baselines achieve only "Minutes"; LingBot-World-Infinity achieves "Hours (Infinite)."
  • Semantic interaction: Baselines have "None" or "Few"; LingBot-World-Infinity achieves "Infinite."
  • Dynamic degree: Tied for "High" with prior LingBot-World; others "Medium."
  • Real-time: All models including this one are real-time.
  • Open-source: HappyOyster and Genie 3 are not open-source; this model is.
  • No quantitative perceptual quality metrics (FID, FVD, PSNR) are reported in the provided text.

Limitations

  • No quantitative video quality or controllability benchmarks are provided; comparison table is purely categorical.
  • The agentic harness evaluation is qualitative (case rollouts only); no task-completion or causal reasoning accuracy metrics are reported.
  • Paper text is truncated; Mode B interaction and multi-player details are incomplete in the available text.
  • Long-horizon stability is demonstrated by a single 1-hour session rather than a systematic stress test across diverse scenes.
  • Distillation quality relative to the teacher model is not quantitatively characterized.

Relevance to Agentic AI / LLM Agents

The Director-Pilot framework is a direct application of the LLM-agent-in-a-harness paradigm to video world modeling—the paper explicitly cites coding assistants (Codex) as the architectural analogue, where a strong LM becomes useful only once scaffolded with state inspection, action execution, and multi-turn goal pursuit. For researchers tracking agentic AI, this work instantiates a multi-agent simulation backbone: the VLM agent handles high-level causal reasoning and planning while a generative model handles low-level execution, a decomposition increasingly common in embodied and robotics agent stacks. The VLM-as-director pattern (scene analysis → event-card proposals → grounded rollouts) is a reusable design motif for any agent system that must reason about world state and produce physically coherent outcomes. The paper also demonstrates that the "harness" philosophy scales beyond code to continuous perceptual environments, broadening its relevance to embodied simulation, game-playing agents, and interactive environment generation for agent training.