ABot-World-0: Infinite Interactive World Rollout on a Single Desktop GPU

🕒 Published (v1): 2026-07-21 15:26 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ABot-World-0 is an action-conditioned video world model for real-time, long-horizon closed-loop interactive simulation on consumer hardware. It combines a multi-source data infrastructure with agent-driven collection, a progressive bidirectional-to-causal distillation pipeline with a novel LongForcing stage, and a co-designed streaming inference stack. The system achieves 720P output at up to 16 FPS with 1.2 s action-to-first-frame latency on a single RTX 5090 within ~19 GiB peak VRAM.

Problem

Prior video world models fail to jointly solve four coupled bottlenecks: (1) acquiring temporally coherent, action-synchronized training data at scale across heterogeneous sources; (2) providing a unified control interface spanning both camera navigation and embodied character control without pose-drift accumulation over long rollouts; (3) preventing autoregressive drift where student self-rollout outputs become the next input context, causing growing train/inference distribution mismatch; and (4) deploying the full generation-and-decoding stack at interactive speeds on practical consumer hardware. Existing systems address at most one or two axes simultaneously.

Method

ABot-World-0 addresses all four bottlenecks through three co-designed components.

Data infrastructure. Training data draws from AAA game recordings (ground-truth API action labels), simulation engines (Unreal Engine and ABot-3DGS 3D Gaussian Splatting for photorealistic scenes), and internet video (COLMAP/ViPE pose-estimated pseudo-labels). WorldExplorer, an agent-driven collection system, autonomously navigates virtual environments via a multi-phase goal selection strategy (unexplored regions → nearby areas → forward-movement fallback with collision detection) with a training-feedback-driven closed loop: per-category performance metrics identify weak scene-action combinations and dynamically adjust collection ratios without manual intervention. Surviving data passes through 14 deterministic quality checks across six dimensions plus VLM-based semantic assessment, yielding structured annotation (action labels, VLM-generated scene captions, semantic and identity tags).

Training pipeline. A bidirectional teacher is first trained on long action-conditioned sequences, exploiting full-horizon attention for high-quality dynamics supervision. A causal student is then progressively distilled via teacher forcing and ODE distillation, preserving visual quality while enforcing the causal constraint required for streaming. LongForcing is a final distribution-matching stage: long student self-rollouts are supervised against an extended-horizon bidirectional teacher at the distribution level, providing corrective signals for the long-horizon rollout contexts that short-horizon distillation objectives cover only weakly. Reference-character memory stores persistent appearance cues for third-person identity consistency. Control uses raw discrete keyboard actions (a fixed, bounded action space) for both scene roaming and character motion, replacing calibrated global camera trajectories to avoid accumulated pose drift.

Deployment stack. Chunk-wise autoregressive generation with progressive visual delivery is paired with a lightweight VAE decoder, memory-aware module scheduling, bounded local-context KV caching, low-bit DiT inference, and efficient low-precision attention.

Key Contributions

• ABot-World-0: unified action-conditioned video world model with a raw keyboard interface covering first-person scene roaming and third-person character control, with reference-character memory for appearance persistence across long rollouts.
• WorldExplorer: training-feedback-driven agent-driven collection system that diagnoses per-category model weaknesses and adaptively rebalances collection effort in a closed loop without manual intervention.
• LongForcing: distribution-matching training stage that supervises long student self-rollouts against an extended-horizon bidirectional teacher, explicitly mitigating accumulated autoregressive drift beyond what local distillation achieves.
• Deployment-oriented streaming stack: 720P at up to 16 FPS with 1.2 s action-to-first-frame latency within ~19 GiB peak VRAM on a single RTX 5090, with low-bit DiT inference and memory-aware scheduling.
• Unified quality pipeline: 14 deterministic checks across six dimensions plus VLM-based semantic assessment, applied to data from three heterogeneous sources with cross-modal alignment error below 33 ms at 30 FPS.

Results

• 720P streaming at up to 16 FPS on a single NVIDIA RTX 5090 desktop GPU across optimized low-bit configurations.
• 1.2 s action-to-first-frame latency measured end-to-end from action receipt to first decoded frame (not sampling speed in isolation).
• ~19 GiB peak VRAM across the optimized low-bit operating envelope.
• Competitive controllability and coherent long-horizon world evolution on WorldRoamBench and extended interactive rollouts (specific numerical scores not reported in the available text).

Limitations

• Performance characterization is limited to a single high-end GPU (RTX 5090); accessibility on lower-VRAM or lower-compute hardware is not evaluated.
• Specific numerical scores on WorldRoamBench are not detailed in the available text, precluding direct quantitative comparison with prior systems.
• Internet video data relies on noisy pose-estimated pseudo-labels (COLMAP/ViPE), which may introduce action annotation errors relative to the ground-truth API labels available for game data.
• Game data is inherently limited in visual style diversity to the available AAA titles used.
• LongForcing depends on a bidirectional teacher operating over an extended but finite horizon; effectiveness at rollout lengths exceeding that horizon is not analyzed.
• The paper is a technical report, not a peer-reviewed venue, so results have not undergone independent validation.

Relevance to Vision-Language Models

VLMs serve a concrete functional role in the data pipeline as scalable semantic oracles: they assess candidate training clips for semantic validity and generate structured scene captions (with game-identifier prefixes and no camera motion leakage) that enable text-conditioned world-model training. This establishes a pattern — VLM-as-annotation-engine for large-scale multimodal video corpora — directly relevant to practitioners building training data pipelines for video-language models. The system's text-conditioning pathway, seeded by VLM-generated captions and semantic/identity tags, also points toward language-steerable world models where VLM outputs become first-class conditioning signals rather than auxiliary metadata, aligning with the broader trajectory of action-language grounding in embodied VLM research.