Skip to content

SimWorld: An Open-ended Simulator for Agents in Physical and Social Worlds

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SimWorld is an Unreal Engine 5-based open-ended simulator designed for developing and evaluating LLM/VLM agents in city-scale physical and social environments. It provides procedural world generation, multimodal perception interfaces, and open-vocabulary action outputs. Experiments with frontier models (GPT-4o, Claude-3.7-Sonnet, Gemini-2.5-Pro, DeepSeek-V3, etc.) reveal systematic gaps in spatial reasoning, traffic-rule compliance, and long-horizon economic decision-making.

Problem

Existing simulators for embodied agent research either use game-like physics (Minecraft), narrow domains (CARLA for driving, AI2-THOR for indoors), small hand-crafted scene sets, or lack native LLM/VLM interfaces (no open-vocabulary actions, no multimodal feedback loops). None support dynamic, city-scale, multi-agent outdoor environments with realistic physics and social norms while being natively compatible with frontier language/vision-language models.

Method

SimWorld is built on a two-tier architecture: an Unreal Engine 5 backend for real-time physics (gravity, momentum, collision) and 3D rendering, and a Python layer for LLM/VLM integration.

World generation: A procedural system generates diverse road networks, building layouts, and urban elements (traffic, pedestrians). A retrieval-augmented LLM-based scene agent accepts natural language edit commands (e.g., "add a red sports car next to the hospital"), queries a semantic scene graph, retrieves or synthesizes 3D assets via text-to-3D models, and inserts them spatially.

Agent interface: Agents receive RGB images, depth maps, segmentation masks, and structured scene graphs. They issue open-vocabulary high-level actions (e.g., "go to the nearest chair and sit down"), which a two-component local action planner decomposes: an LLM-based parser grounds intentions, and a rule-based or VLM-based executor handles low-level environment interaction.

Case studies: - Case 1 (physical reasoning): Vision-based urban navigation with static/dynamic obstacles; agents choose from discrete actions (step, turn, change view, change waypoint) and are evaluated with and without the local action planner. - Case 2 (social reasoning): Multi-agent delivery economy with energy, economic, and order-sharing systems. Agents (20 per model, 5000 steps each) use ReAct prompting and a two-tiered action space (bid, pick up, deliver, share, purchase, adjust speed). Big Five personality traits are injected via prompt to study behavioral emergence.

Key Contributions

  • Open-ended, language-controllable procedural city-scale simulator built on UE5 with realistic physics and social norms (traffic signals, personal space, pedestrian dynamics).
  • Native LLM/VLM agent interface: multimodal inputs (RGB, depth, segmentation, scene graph) + open-vocabulary high-level action outputs with layered execution.
  • Hierarchical local action planner that decouples high-level reasoning from low-level locomotion, shown by ablation to substantially improve navigation success.
  • Two benchmarking case studies spanning embodied physical reasoning and strategic multi-agent social reasoning with quantitative metrics (SR, CC, RVR, STR, NDC, profit, energy efficiency, sharing count).
  • Comparative evaluation of 9+ frontier models revealing distinct failure modes and behavioral patterns across model families.

Results

Case 1 — Navigation with local action planner (static obstacles): - GPT-4o: SR 93.33% across Easy/Medium/Hard; RVR 67.86%/100%/100% — high success but consistent red-light violations. - Claude-3.7-Sonnet: SR 86.67%/96.67%/93.33%; RVR 46.43%/100%/0% — efficient (NDC ~2.07–2.12) but also ignores red lights at scale. - Gemini-2.5-Pro: SR 96.7%/83.3%/80.0%; lowest collision counts (CC-S 1.45–3.08) but higher stuck rate. - Rule-based baseline: SR 66.73%/63.33%/40.0%; 0% RVR on Easy, 100% on Medium/Hard.

Without local action planner (ablation): Gemini-2.5-Pro best autonomous planning; Claude-3.7-Sonnet efficient but less safe; GPT-4o fails to adapt.

Case 2 — Multi-agent delivery (profit, mean ± std over 3 rounds): - DeepSeek-V3: 69.48 ± 16.77 profit, 2.10 avg successful orders. - Claude-3.5-Sonnet: 69.07 ± 20.69 profit, highest sharing count (11.33) and investment count (9.00) — highest variance. - GPT-4o: 43.91 ± 14.16 profit. - Gemini-2.5-Flash: 42.42 ± 3.10 profit — most stable (lowest std). - GPT-4o-mini: 0 profit across all metrics.

Model competition (12 models, 24 agents, 1000 steps): DeepSeek-Prover-V2 and Qwen3-32B highest head-to-head win rates via lower bid prices; Claude-3.7-Sonnet and Gemini-2.5-Flash show broad bid distributions enabling flexible strategy.

Persona ablation: Conscientiousness correlates positively with bid win rate (+0.65) and order pickup frequency (+0.66); Openness negatively correlates with delivery actions (−0.70 with deliver count).

Limitations

  • Paper is truncated; full limitations section is cut off.
  • Graphical rendering disabled during Case 2 simulation, removing visual perception from the social reasoning evaluation.
  • Scene and social complexity is still urban/delivery-focused; generalization to other physical domains (indoor, non-urban) not demonstrated.
  • LLM/VLM agents consistently violate traffic rules (100% RVR in several conditions), suggesting that passive visual perception of traffic signals is not reliably acted upon — a fundamental gap not addressed by the simulator itself.
  • Personality injection is prompt-based only; no learned or grounded persona modeling.
  • Evaluation scale is limited (20 agents per model, 3 seeds), which may not capture full behavioral variance.

Relevance to Vision-Language Models

SimWorld provides a rigorous, physics-grounded benchmark that directly stresses VLM capabilities — specifically multimodal scene understanding, grounded spatial reasoning, and closed-loop perception-action alignment — revealing concrete failure modes (e.g., models perceive but do not act on red-light signals) that pure VQA or captioning benchmarks cannot surface. The open-vocabulary action interface and layered input modalities (RGB, depth, segmentation, scene graph) make it a natural testbed for studying how VLMs integrate heterogeneous visual representations into sequential decision-making. The social reasoning case study extends VLM evaluation into multi-agent economic settings where language-mediated negotiation, persona conditioning, and strategic planning interact, complementing existing single-turn VLM benchmarks. For researchers tracking VLMs, SimWorld offers both a diagnostic tool for current model limitations and a training environment for future embodied VLM agents.