Skip to content

SimWorld-Robotics: Synthesizing Photorealistic and Dynamic Urban Environments for Multimodal Robot Navigation and Collaboration

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SimWorld-Robotics (SWR) is a photorealistic urban embodied AI simulation platform built on Unreal Engine 5 that procedurally generates city-scale environments populated with dynamic pedestrians and traffic. It introduces two benchmarks—multimodal robot navigation (SimWorld-MMNav) and multi-robot search (SimWorld-MRS)—that expose critical failures in state-of-the-art VLMs. Even the best zero-shot models achieve 0% full-task success rate on the navigation benchmark, and fine-tuning a 7B VLM yields only 4% SR.

Problem

Existing embodied AI simulators focus on indoor/household settings; outdoor urban simulators (CARLA, MetaDrive, MetaUrban) lack photorealism, procedural scalability, flexible multi-agent control, or realistic pedestrian/traffic dynamics. This leaves a gap in evaluating robot foundation models on the key capacities needed for real-world urban deployment: 3D spatial reasoning at scale, safe navigation among dynamic agents, long-horizon planning, and grounded multi-robot communication.

Method

SWR is built on Unreal Engine 5 and extends SimWorld with a four-stage procedural city generation pipeline: (1) road network growth via priority-queue branching with intersection validation; (2) collision-aware building placement with greedy gap-filling; (3) contextual street element placement; (4) dynamic traffic/pedestrian injection. It supports asynchronous multi-agent control via a centralized buffer updated at 0.01 s intervals, allowing simultaneous independent agent actions. Observation spaces include RGB, depth, and semantic segmentation; action spaces cover continuous vehicle control and discrete robot translation/rotation.

SimWorld-MMNav: A robot follows paired language+visual-hint instructions through city blocks (avg. 500 m, 250 steps), evaluated on Success Rate, Subtask SR, and Distance Progress. Hard setting adds static obstacles, pedestrians, and traffic with collision/red-light-violation metrics.

SimWorld-MRS: Two robots (one map-aware, one naive) must localize each other via natural-language dialogue and physically navigate to meet, evaluated by Collaborative Success Rate and Task Progress.

SimWorld-20K: 20K training steps from 200 oracle A* trajectories across 100 2 km² cities, used to fine-tune QwenVL2.5-7B.

Key Contributions

  • SWR simulator: Unreal Engine 5 platform with procedural city generation producing environments 100Ă— larger in area than MetaUrban (avg. 2 km²) and supporting 26 human action types vs. ≤2 in prior work.
  • SimWorld-MMNav benchmark: multimodal (vision+language) instruction-following navigation in photorealistic dynamic urban scenes.
  • SimWorld-MRS benchmark: multi-robot collaborative search requiring grounded natural-language communication and physical navigation.
  • SimWorld-20K dataset: 20K-step training set with oracle trajectories for fine-tuning VLMs on long-horizon city-scale navigation.
  • Systematic evaluation identifying failure modes of current VLMs on urban embodied tasks.

Results

SimWorld-MMNav (easy): - All zero-shot open-source VLMs (QwenVL2.5-7B/72B, Gemma 3 27B, InternVL3 78B): SR = 0% - GPT-4o and Gemini 2.5 Flash: SR = 0%; Gemini achieves highest Distance Progress (31.29%) - GPT-o3-pro (reasoning model): SR = 8.3%, Subtask SR = 46.35%, Distance Progress = 39.46% - Fine-tuned QwenVL2.5-7B: SR = 4.0%, Subtask SR = 52.45% (+35.59), Distance Progress = 53.63% (+45.81) - HybridGPT (GPT-4o + A*): SR = 0%, Distance Progress = 27.24% - VLA-RL baseline: SR = 0%, Distance Progress = 22.79%

SimWorld-MMNav (hard): - GPT-4o: SR = 2.08%, 1.92 static collisions, 10.37 dynamic collisions, 3.02 red-light violations - Gemini 2.5 Flash: SR = 0%, lower dynamic collisions but 7.875 red-light violations

SimWorld-MRS: - GPT-4o + Oracle Planner: CSR = 65.00%, Task Progress = 76.90% - Gemini 2.5 Flash + Oracle Planner: CSR = 54.55%, Task Progress = 75.84% - GPT-4o + RoCo: CSR = 33.33%, Task Progress = 22.93% - QwenVL2.5-72B + RoCo: CSR = 11.11%, Task Progress = 35.94%

Limitations

  • Simulator covers outdoor environments only; no indoor scenes.
  • Human agent action space, while richer than prior work (26 actions), is still limited; no learned motion generation.
  • Training set uses oracle A* traces, limiting fine-tuned model robustness (no corrective demonstrations or RL).
  • Reasoning models (GPT-o3) have high inference latency, making them unsuitable for real-time hard-setting evaluation.
  • Pedestrian/traffic simulation uses simple probabilistic routing and PID controllers, not learned or socially-aware models.

Relevance to Vision-Language Models

SWR provides the first large-scale, photorealistic urban benchmark for stress-testing VLMs as embodied agents, directly quantifying their failures in 3D spatial reasoning, visual grounding, long-horizon planning, and grounded multi-agent communication—capabilities that indoor benchmarks do not probe. The finding that zero-shot frontier VLMs (GPT-4o, Gemini 2.5 Flash) achieve 0% full-task success while fine-tuning a 7B model on synthetic oracle data yields only 4% SR exposes a fundamental gap between VLM perception/reasoning and embodied urban deployment. For VLM researchers, the failure taxonomy (visual grounding, 3D spatial estimation, pragmatic instruction interpretation, memory/planning) offers concrete targets, and the SimWorld-20K dataset provides a scalable synthetic training resource for future work on VLM-based robot navigation.