SimWorld-Robotics: Synthesizing Photorealistic and Dynamic Urban Environments for Multimodal Robot Navigation and Collaboration¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

SimWorld-Robotics (SWR) is an Unreal Engine 5-based simulation platform that procedurally generates photorealistic, dynamic urban environments for embodied AI research. It introduces two benchmarks—multimodal instruction-following navigation (SimWorld-MMNav) and multi-robot search (SimWorld-MRS)—and demonstrates that state-of-the-art VLMs including GPT-4o and Gemini 2.5 Flash fail to achieve meaningful success on either task.

Problem¶

Existing embodied AI simulators and benchmarks are overwhelmingly indoor/household-focused; outdoor urban simulators (CARLA, MetaDrive, MetaUrban) lack either photorealism, procedural generation, multi-agent support, or realistic pedestrian/traffic dynamics. There is no comprehensive benchmark testing VLMs on long-range urban navigation with social navigation constraints and multi-robot grounded communication simultaneously.

Method¶

SWR is built on Unreal Engine 5 and extends the SimWorld base with: - Procedural city generation: a four-stage pipeline (road network via priority-queue growth, building placement via collision-aware greedy sampling, street element placement, dynamic traffic/pedestrian integration) producing infinite 2+ km² cities. - Asynchronous multi-agent control: a centralized observation buffer updated at 0.01s intervals; agents act independently without synchronization barriers. - Rich embodied agents: humans (26 action types), scooter and quadruped robots, vehicles with PID-controlled dynamics. - SimWorld-MMNav benchmark: robot follows paired language+image instructions across 500m episodes in procedurally generated cities; evaluated with Success Rate, Subtask SR, and Distance Progress; hard setting adds static collision, dynamic collision, and traffic light violation metrics. - SimWorld-MRS benchmark: two robots (one map-aware, one naive) must localize each other and physically meet via natural-language communication in 2.5 km² environments; evaluated with Collaborative Success Rate and Task Progress. - SimWorld-20K dataset: 200 oracle A*-generated trajectories across 100 cities, 20K steps, avg. episode length >2.5 km, used for supervised fine-tuning.

Key Contributions¶

SWR simulator: photorealistic procedural urban environments at city-scale (100× larger area than MetaUrban), with asynchronous multi-agent control and dynamic traffic/pedestrian simulation.
SimWorld-MMNav benchmark: multimodal (vision+language) instruction-following navigation with dynamic obstacles and traffic-rule compliance.
SimWorld-MRS benchmark: multi-robot search requiring grounded spatial communication and physical co-navigation in large unknown environments.
SimWorld-20K: large-scale training dataset enabling fine-tuning of VLMs for city-scale navigation.
Systematic baseline evaluation exposing critical failure modes of current VLMs in urban embodied settings.

Results¶

SimWorld-MMNav (easy setting): - All zero-shot open-source VLMs (QwenVL2.5-7B/72B, Gemma3-27B, InternVL3-78B): 0% SR - GPT-4o: 0% SR, 33.07% Subtask SR, 15.60% Distance Progress - Gemini 2.5 Flash: 0% SR, 37.06% Subtask SR, 31.29% Distance Progress - GPT-o3: 5.0% SR; GPT-o3-pro: 8.3% SR (only non-zero among zero-shot) - QwenVL2.5-7B fine-tuned on SimWorld-20K: 4.0% SR, 52.45% Subtask SR, 53.63% Distance Progress (+45.81pp over base)

SimWorld-MMNav (hard setting, selected models): - GPT-4o: 2.08% SR with 1.92 static collisions, 10.37 dynamic collisions, 3.02 red-light violations - Gemini 2.5 Flash: 0% SR, fewest dynamic collisions but highest red-light violations (freezes in intersections)

SimWorld-MRS: - GPT-4o + Oracle Planner: 65.0% CSR, 76.90% Task Progress - Gemini 2.5 Flash + Oracle Planner: 54.55% CSR - GPT-4o + RoCo: 33.33% CSR, 22.93% Task Progress - QwenVL2.5-72B + RoCo: 11.11% CSR

Limitations¶

Simulator covers only outdoor environments; no indoor scenes.
Human agent action space (26 actions) is still limited compared to real-world diversity; no motion-generation-based animation.
SimWorld-20K training data is oracle-traced A* trajectories, which limits robustness of fine-tuned models (no corrective/recovery demonstrations).
Reasoning models (GPT-o3) are unsuitable for the hard real-time setting due to inference latency.
Absolute success rates remain very low even after fine-tuning, indicating the benchmark may be too hard for current systematic improvement measurement.

Relevance to Agentic AI / LLM Agents¶

SWR directly stress-tests the core agent loop—perception, grounding, planning, action, and multi-agent communication—in a realistic, long-horizon setting that exposes failures invisible in tabletop or indoor benchmarks. The SimWorld-MRS benchmark is particularly relevant to agentic AI: it requires two LLM-driven agents to maintain spatial memory, generate grounded natural-language descriptions, interpret each other's partial observations, and jointly plan rendezvous—a prototype of real-world multi-agent coordination. The consistent failure of SOTA proprietary VLMs (GPT-4o at 0% SR on easy navigation) quantifies how far current foundation models are from robust urban embodied agency, establishing SWR as a high-bar evaluation target for the agentic robotics pipeline.