Virtual Community: An Open World for Humans, Robots, and Society¶

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; Virtual Community: open world simulator for human-robot-AI coexistence study

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

Virtual Community is an open-world simulation platform that unifies human avatars and heterogeneous robots in large-scale, real-world-aligned 3D environments generated from geospatial data. It uses LLMs to auto-populate these worlds with socially-grounded agent communities and proposes two benchmarks—Community Planning and Community Robot—to evaluate multi-agent reasoning and low-level cooperation. No existing simulator supports human-robot co-simulation at this scale with realistic physics.

Problem¶

Existing embodied AI simulators either target robots or human-like agents in isolation, handle only small populations with limited physics interactions, or lack scalable open-world (indoor + outdoor) environments. The result is that community-scale human-robot coexistence—with realistic social dynamics and physical interactions—cannot currently be studied or benchmarked.

Method¶

Scene Generation. The pipeline ingests Google 3D Tiles geospatial data, then performs: (1) geometry reconstruction—terrain via bilinear interpolation from sparse elevation points, buildings from OpenStreetMap primitives matched to Google Tiles geometry; (2) texture enhancement—Stable Diffusion 3 inpainting to repair baked textures, then refinement with street-view imagery; (3) object placement—generative (SD + One-2-3-45 mesh lifting) for simple objects, retrieval for complex ones; (4) place/transit annotation via Google Maps Places and OSM. 35 annotated urban scenes worldwide were generated.

Agent Community Generation. GPT-4o is prompted with scene metadata (place names, types, functions) and avatar appearance attributes to generate character profiles (name, age, occupation, personality values, hobbies) and social relationship networks organized as named groups with designated activity locations. A grounding validator checks that all referenced places exist and re-prompts for correction on failure.

Unified Simulation. The Genesis physics engine hosts both components. Human avatars use SMPL-X skeletons with pose vectors \(J \in \mathbb{R}^{162}\) and global translation/rotation \(T, R \in \mathbb{R}^{3}\), driven by >2,000 Mixamo motion clips. Five robot types (drone, quadruped, humanoid, wheeled, mobile manipulator) share the same simulation loop at different control frequencies. Daily schedules are LLM-generated per agent, including explicit commute-time modeling.

Community Planning Baselines. All planners share a hierarchical framework with a common low-level navigation stack: egocentric RGB-D → 0.1 m point cloud → 0.5 m 2D occupancy map → A* pathfinding, re-planned every step. High-level planners evaluated: Random, Heuristic (finite-state automaton), MCTS, and LLM Planner (GPT-4o via CoELA-style modular framework with natural-language inter-agent communication).

Key Contributions¶

Open-source unified simulation platform for co-located human avatars and heterogeneous robots, built on Genesis.
Automated real-world-grounded scene generation pipeline (geospatial data + generative models → interactive 3D urban scenes at arbitrary locations).
LLM-driven agent community generation with validated scene-grounded character profiles and social network graphs.
Community Planning Challenge: three cooperative assistant tasks (Carry, Delivery, Search) and one competitive social-influence task, averaged over 24 scenes.
Community Robot Challenge: multi-robot cooperation across indoor/outdoor environments (details truncated in provided text).
Ablation demonstrating the necessity of explicit distance modeling for open-world planners.

Results¶

Community Planning Challenge results averaged over 24 scenes (Success Rate SR↑, Time Ts↓, Human Following Rate HR↑):

1-assistant setting: LLM Planner 47.2% avg SR; MCTS Planner 42.4%; Heuristic 42.1%; Random 0%.
LLM Planner leads on Search (70.1% SR, \(T_s\)=1339.0) but underperforms on Carry (29.9% SR vs. Heuristic 34.7%).
2-assistant setting: Heuristic 54.6% avg SR; LLM Planner 50.6%; MCTS 44.9%.
LLM Planner again leads Search (77.8% SR, \(T_s\)=1141.3); Heuristic leads Carry (52.8% SR).
Distance Modeling ablation (Table 2): Removing explicit distance modeling drops MCTS from 42.4% to 29.0% avg SR (1-assistant); LLM Planner drops from 47.2% to 44.4%—a smaller but non-trivial decline. MCTS is far more sensitive than the LLM Planner to this feature.
No single method dominates across all tasks; LLM Planner is strongest on spatially unconstrained search but weaker on constrained following/carrying tasks.

Limitations¶

No single baseline achieves strong performance across all task types, suggesting the benchmarks are genuinely hard but also that evaluation of more capable agents is deferred to future work.
Avatar physics uses kinematic attachment (not full dynamics) for object manipulation and vehicle riding, limiting realism of physical interactions.
Community Robot Challenge results are not available in the provided text.
GPT-4o is used for agent community generation and LLM planning, creating a cost and reproducibility dependency on a proprietary model.
The 1500-step simulation cap causes many tasks to time out (Ts = 1500.0), potentially masking failure modes from planning versus execution.
Scene diversity is bottlenecked by availability and quality of Google 3D Tiles and OSM data; low-interactivity aerial-reconstruction artifacts require expensive cleanup.

Relevance to Agentic AI / LLM Agents¶

Virtual Community is directly relevant as a large-scale testbed for LLM-driven multi-agent planning: the LLM Planner baseline instantiates a GPT-4o agent that perceives environment state, generates hierarchical subplans, and communicates in natural language with peer agents—a canonical agentic architecture. The agent-community generation pipeline demonstrates how LLMs can bootstrap entire societies of agents with consistent personas, relationships, and schedules from minimal structured input, extending the generative-agents (Park et al. 2023) paradigm to physically grounded 3D environments. The platform's distance-modeling ablation is a concrete finding about a failure mode of LLM and search-based planners in open worlds where spatial cost matters. For researchers building or evaluating LLM agents in embodied or social settings, this work establishes a more realistic evaluation regime than household-scale or gridworld benchmarks.