Skip to content

SimWorld-Robotics: Synthesizing Photorealistic and Dynamic Urban Environments for Multimodal Robot Navigation and Collaboration

🕒 Published (v1): 2025-12-10 20:04 UTC · Source: Arxiv · Venue: NEURIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SimWorld-Robotics (SWR) is a photorealistic urban simulation platform built on Unreal Engine 5 that procedurally generates infinite city-scale environments with dynamic pedestrians and traffic. It introduces two benchmarks—multimodal instruction-following navigation (SimWorld-MMNav) and multi-robot search (SimWorld-MRS)—and shows that all current SOTA VLMs fail or barely succeed on both tasks. Fine-tuning QwenVL2.5-7B on the accompanying SimWorld-20K dataset yields the only non-zero full-task success rate.

Problem

Embodied AI simulators overwhelmingly target indoor/household scenarios. Existing outdoor simulators (CARLA, AirSim, MetaUrban) lack some combination of photorealism, procedural city generation, flexible multi-robot/multi-agent control, and realistic dynamic social elements (pedestrians obeying traffic rules). There are no benchmarks that jointly evaluate 3D spatial reasoning, multimodal instruction grounding, long-range navigation with traffic, and grounded multi-robot communication in realistic urban environments.

Method

SWR extends Unreal Engine 5 with a four-stage procedural city generation pipeline: (1) priority-queue-based road network growth with intersection validation, (2) collision-aware greedy building placement, (3) contextual street element seeding (trees, benches, parked vehicles), (4) dynamic traffic/pedestrian injection governed by PID-controlled vehicles and probabilistic routing at intersections. It supports three agent types (humans with 26 actions, vehicles, quadruped/scooter robots) under an asynchronous multi-agent control framework with a centralized observation buffer at 0.01 s update intervals. Observations include RGB, depth, and semantic segmentation. Two benchmarks are built on SWR: - SimWorld-MMNav: sequential multimodal instruction following (language + visual hint pairs) across procedurally generated 500 m average routes in cities averaging 2 km²; evaluated at easy (no obstacles) and hard (dynamic pedestrians + traffic) difficulty. - SimWorld-MRS: two robots (one map-aware, one blind) must localize each other and physically meet via grounded natural-language dialogue; environments average 2.5 km² with 20 landmarks, 576 m mean inter-robot spawn distance.

Baselines use VLMs (GPT-4o, GPT-o3, Gemini 2.5 Flash, QwenVL 2.5, Gemma 3, InternVL 3) under ReAct, plus a hybrid GPT-4o + A* planner, an RL baseline (VLA-RL with DeBERTav3 + DINOv2), and for MRS a RoCo-style collaborative planning protocol.

Key Contributions

  • SimWorld-Robotics (SWR) simulator: procedural infinite city generation, photorealistic UE5 rendering, asynchronous multi-agent control, 26 human actions, dynamic traffic+pedestrian systems.
  • SimWorld-MMNav benchmark: first urban navigation benchmark jointly evaluating multimodal grounding, 3D spatial reasoning, obstacle/traffic avoidance, and long-horizon (500 m, 250-step) navigation.
  • SimWorld-MRS benchmark: first benchmark requiring two robots to physically navigate to meet each other via grounded dialogue in a large (2.5 km²) urban environment.
  • SimWorld-20K dataset: 20K training steps from 200 oracle A*-traced trajectories across 100 procedurally generated 2 km² cities.
  • Systematic baseline evaluation revealing fundamental gaps in current VLMs for urban embodied tasks.

Results

SimWorld-MMNav (easy) - All zero-shot non-reasoning VLMs achieve 0% full-task Success Rate (SR); GPT-o3-pro reaches 8.3% SR. - Fine-tuned QwenVL2.5-7B: 4.0% SR, 52.45% Subtask SR (+35.59 pp), 53.63% Distance Progress (+45.81 pp) — only model with non-zero SR. - Gemini 2.5 Flash: highest zero-shot Distance Progress (31.29%) among non-reasoning models. - VLA-RL (RL baseline) fails to beat zero-shot LLMs.

SimWorld-MMNav (hard, subset) - GPT-4o: 2.08% SR, 1.92 static collisions, 10.37 dynamic collisions, 3.02 red-light violations. - Gemini 2.5 Flash: 0% SR but lowest dynamic collisions (4.29); high red-light violations (7.875) due to freezing inside intersections. - Reasoning models excluded from hard setting due to inference latency incompatibility with real-time traffic.

SimWorld-MRS - GPT-4o + Oracle Planner: 65.0% CSR, 76.90% Task Progress (upper bound). - GPT-4o + RoCo: 33.33% CSR, 22.93% Task Progress. - QwenVL2.5-72B + RoCo: 11.11% CSR, 35.94% Task Progress.

Limitations

  • Simulator covers only outdoor environments; no indoor scenes.
  • Human agent action space, though larger than prior simulators (26 actions), is still limited; future work plans motion generation models for richer behavior.
  • SimWorld-20K training uses oracle A* traces, limiting robustness of fine-tuned policies (no corrective or RL-based demonstrations).
  • Reasoning models (GPT-o3) are too slow for real-time traffic avoidance in the hard benchmark setting.
  • Fine-tuned model achieves only 4% SR even on the easy split, indicating the benchmark remains largely unsolved.

Relevance to Agentic AI / LLM Agents

SWR directly stress-tests LLM/VLM-based agents in the loop-closed, embodied regime: agents must ground multimodal instructions, maintain spatial memory across hundreds of steps, and produce executable low-level actions—all capabilities central to agentic AI research. The finding that GPT-o3-pro achieves only 8.3% SR on the easy split despite strong chain-of-thought reasoning quantifies the frontier gap between language-model reasoning and real embodied agency. The multi-robot search benchmark specifically evaluates grounded inter-agent communication and joint planning, a canonical multi-agent LLM scenario now tested against physical navigation ground-truth rather than text-only protocols. SimWorld-20K also provides a scalable synthetic data pipeline relevant to training more capable embodied agents.