Skip to content

SimWorld: An Open-ended Simulator for Agents in Physical and Social Worlds

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SimWorld is an Unreal Engine 5-based open-ended simulator for developing and evaluating LLM/VLM agents in physically and socially realistic urban environments. It supports procedural city-scale world generation, natural language scene editing, and a rich multimodal agent interface with open-vocabulary action outputs. Experiments with frontier models (GPT-4o, Claude-3.7-Sonnet, Gemini-2.5-Pro, DeepSeek-V3, etc.) reveal distinct reasoning patterns and failure modes across physical navigation and multi-agent delivery tasks.

Problem

Existing simulators for embodied agent development are limited by hand-crafted scenes, simplified game-like physics (e.g., Minecraft's discrete block mechanics), narrow domain scope (household or driving only), and lack of native LLM/VLM interfaces. Social sandboxes that do exist (e.g., Virtual Village) are scripted and small-scale. No prior platform combines realistic physical/social dynamics, language-driven procedural generation, and a natural-language action interface in a single open-ended urban environment suitable for benchmarking frontier models.

Method

SimWorld is built on a two-tier architecture: an Unreal Engine 5 backend for physics rendering and simulation, and a Python layer for procedural generation, agent interfaces, and scenario construction.

  • World generation: Procedural system generates diverse city-scale environments (road networks, building layouts, traffic, pedestrians). Natural language commands are processed by a retrieval-augmented LLM scene agent that queries a semantic scene graph, retrieves assets from a library, and synthesizes novel 3D assets via text-to-3D generation when needed.
  • Agent interface: Multimodal inputs include RGB images, depth maps, segmentation masks, and structured scene graphs. Agents issue open-vocabulary high-level actions (e.g., "go to the nearest chair and sit down"); a two-component local action planner—an LLM-based parser plus a rule-based or VLM-based executor—decomposes these into low-level simulator commands.
  • Case Study 1 (physical reasoning): Vision-based urban navigation with static/dynamic obstacles; agents choose from a discrete set (step, turn, change view, change waypoint) and are evaluated with and without A*-guided local action planner.
  • Case Study 2 (social reasoning): Multi-agent delivery economy with energy/economic/order-sharing systems; agents use ReAct prompting to bid, pick up, deliver, share orders, and invest (e.g., purchase scooters). Ablations cover model competition (24 agents, 12 models), environment configuration (order scarcity, initial capital), and Big Five personality traits injected via prompt.

Key Contributions

  • Open-ended UE5-based simulator combining realistic physics (gravity, momentum, inertia) and social norms (traffic signals, personal space) at city scale.
  • Language-controllable procedural environment generation with on-the-fly scene editing and text-to-3D asset synthesis.
  • Native LLM/VLM agent interface with multimodal observations and open-vocabulary high-level action space bridged to low-level execution.
  • Two complementary evaluation case studies spanning physical (single-agent navigation) and social (multi-agent economic competition/collaboration) reasoning.
  • Empirical characterization of frontier model behaviors, failure modes, and personality-driven strategy divergence across 9+ models.

Results

Case Study 1 — Navigation (with local action planner, static obstacles): - GPT-4o: SR 93.33% across Easy/Medium/Hard; high red-light violation rate (67–100%) - Claude-3.7-Sonnet: SR 86.67/96.67/93.33% (E/M/H); 100% red-light violation on Medium/Hard - Gemini-2.5-Pro: SR 96.70/83.30/80.0%; fewest collisions; most cautious - Rule-based baseline: SR 66.73/63.33/40.0%; 100% red-light violation at Medium/Hard - Without local planner (Medium): Gemini-2.5-Pro best at autonomous planning; GPT-4o fails without predefined waypoints; Claude-3.7-Sonnet efficient but less safe

Case Study 2 — Multi-agent delivery (5000 steps, 20 agents per model): - DeepSeek-V3: highest avg profit 69.48 (std 16.77); Claude-3.5-Sonnet: 69.07 (std 20.69) - Gemini-2.5-Flash: moderate profit 42.42 with lowest std (3.10) — most stable - GPT-4o-mini: 0.00 profit across all metrics (complete failure) - Personality ablation: Conscientiousness correlates positively with bid win rate (+0.65) and order pickup (+0.66); Openness correlates negatively with delivery count (−0.10) and energy efficiency (−0.44) - Model competition: DeepSeek-Prover-V2 and Qwen3-32B achieve highest head-to-head win rates via lower bid prices; Claude-3.7-Sonnet and Gemini-2.5-Flash show broadest bid distributions

Limitations

  • Paper text is truncated; stated limitations section is incomplete.
  • Graphical rendering disabled during multi-agent delivery experiments, removing visual perception from that task.
  • Social/physical simulation is urban-centric; generalization to non-city environments is undemonstrated.
  • Agent interface relies on an LLM-based action parser (e.g., GPT-4o), introducing a dependency on external model calls for low-level grounding.
  • Evaluation covers only two task types; broader task diversity (manipulation, long-term planning across days) is not demonstrated.
  • High token cost per step (~7000 tokens × 2 API calls per agent per step) limits scale of evaluation.
  • All models still exhibit red-light violations; ablation confirms 100% violation rate when red lights are explicitly shown, revealing a systematic active-attention failure.

Relevance to Agentic AI / LLM Agents

SimWorld directly advances the embodied agent evaluation infrastructure that the field needs: it provides a scalable, physically grounded, socially complex environment where LLM agents must plan, perceive, and act over long horizons—tasks fundamentally harder than math or code benchmarks. The open-vocabulary action interface and hierarchical planner design are directly applicable to real-world agent architectures. The multi-agent delivery task operationalizes emergent behaviors (cooperation, competition, resource strategy) that are central to multi-agent LLM systems research. The personality-driven behavioral divergence findings are relevant to agent persona and alignment work, and the systematic red-light violation results highlight a critical gap between passive visual recognition and active attention in current VLMs.