Skip to content

WorldGym: World Model as An Environment for Policy Evaluation

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Percy Liang (Stanford); ICLR 2026; world-model as policy evaluation environment

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WorldGym is a world-model-based policy evaluation environment that uses an autoregressive, action-conditioned video diffusion model to simulate robot rollouts, with a VLM (GPT-4o) providing reward signals from generated frames. Policy success rates inside WorldGym achieve Pearson \(r = 0.78\) correlation with real-world success rates across three VLA families, differing by only 3.3% on average. It enables cheap, reproducible OOD evaluation without physical robot setup.

Problem

Real-world robot policy evaluation is expensive, slow, and hard to scale, while handcrafted simulators suffer from a sim-to-real gap—especially for complex dynamics like soft-object manipulation. Existing offline policy evaluation (OPE) methods assume full observability or ground-truth state access, which is impractical for real robots operating from image observations.

Method

WorldGym trains a latent Diffusion Transformer (DiT) on Open-X Embodiment data using Diffusion Forcing, which enables causal autoregressive frame generation. Per-frame robot action vectors are linearly projected to model dimension and injected via AdaLN-Zero modulation with classifier-free guidance (random action dropout during training ensures the model is fully action-controllable). At inference, the model runs an iterative loop: the current observation \(o_t\) is fed to a policy \(\pi\), which outputs an action chunk \(a_\text{pred}\); the world model predicts one new frame per action in \(a_\text{pred}\) using a causal temporal attention mask. The diffusion horizon is flexibly matched to the policy's action chunk size \(|a_\text{pred}|\), avoiding wasted compute from fixed-horizon designs (e.g., Cosmos's fixed 16-frame window). GPT-4o receives the full generated rollout plus the language instruction to output a binary or partial-credit reward \(\hat{R}\).

Formally, the estimated policy value is: $\(\hat{\rho}(\pi) = \mathbb{E}\bigl[\hat{R}([o_0,\ldots,o_H], g) \mid s_0, g \sim \mathcal{G},\; a \sim \pi(o,g),\; o' \sim \hat{T}(o,a)\bigr]\)$

OOD evaluation is enabled by injecting edited initial frames (via image generation models) or modified language instructions, requiring no re-training.

Key Contributions

  • Single world model trained on diverse Open-X Embodiment data that generalizes across robot morphologies and tasks for policy evaluation.
  • Flexible diffusion horizon alignment with policy action chunk sizes, enabling efficient rollouts from a single checkpoint.
  • Demonstration that WorldGym policy value estimates correlate strongly (\(r=0.78\)) with real-world success rates across RT-1-X, Octo, and OpenVLA.
  • Preservation of relative policy rankings across versions (Octo-Small vs. Octo-Base), sizes (OpenVLA 7B dev vs. final), and training checkpoints (evaluated at 2K–60K steps).
  • OOD evaluation framework revealing concrete VLA failure modes: shape-insensitive object grasping, 2D/3D distractor confusion, and near-chance performance on fine-grained classification.
  • All WorldGym rollouts complete in under one hour on a single GPU vs. days for real-world evaluation.

Results

  • Pearson \(r = 0.78\) (\(p < 0.001\)) between per-task WorldGym success rates and real-world success rates across 17 Bridge tasks × 3 policies.
  • Mean success rate gap of 3.3% on average: RT-1-X 18.5% (real) vs. 15.6% (WorldGym); Octo 20.0% vs. 23.8%; OpenVLA 70.6% vs. 67.4%.
  • Relative rankings RT-1-X < Octo < OpenVLA preserved in WorldGym, matching real-world order.
  • Larger/newer model variants (Octo-Base > Octo-Small; OpenVLA 7B > OpenVLA v0.1) ranked correctly inside WorldGym.
  • Training-step ordering preserved for both a video prediction policy (UniPi) and a diffusion policy (DexVLA) evaluated at 4 checkpoints each.
  • OOD distractor experiment: all three VLAs degrade with random OOD objects; OpenVLA is most robust.
  • OpenVLA achieves perfect color-classification score ("pick red"/"pick blue"); Octo and RT-1-X perform at chance.
  • OpenVLA grabbed a carrot displayed on a laptop screen in 15% of distractor trials (2D/3D confusion).

Limitations

  • Realistic object interaction (deformable objects, contact dynamics) remains visually challenging for the world model.
  • Per-task absolute success rates in WorldGym still diverge from real-world values even if mean and ranking are preserved.
  • OOD image editing requires an external image generation model (Nano Banana / Google 2025).
  • VLM reward (GPT-4o) accuracy is validated in an appendix but introduces its own error surface; partial-credit criteria require manual specification.
  • Evaluation is restricted to short-horizon manipulation tasks on Bridge and Google Robot; long-horizon tasks and legged locomotion are untested.
  • Compounding error from autoregressive rollouts over hundreds of steps is acknowledged but not quantified.

Relevance to Vision-Language Models

WorldGym positions VLMs as learned reward models within a generative simulator loop—GPT-4o judges task completion from generated video frames rather than ground-truth state, making VLM reward quality a direct bottleneck for evaluation accuracy. The paper also benchmarks VLA (Vision-Language-Action) policies that embed VLMs as backbones, and the OOD experiments directly surface VLM-inherited failure modes (shape discrimination, 2D/3D confusion) in downstream robot policy behavior. For researchers tracking VLMs, this establishes a scalable methodology for probing VLM grounding quality without physical robots—any improvement in VLM spatial/shape reasoning would directly improve both the policy being evaluated and the reward signal assessing it.