FreeAskWorld: An Interactive and Closed-Loop Simulator for Human-Centric Embodied AI¶
🕒 Published (v1): 2025-11-17 15:58 UTC · Source: Arxiv · Venue: AAAI 2026 · link
Ask a follow-up
Open an assistant pre-loaded with this paper's context.
💬 Ask ChatGPT✦ Ask Claude
TL;DR¶
FreeAskWorld is a closed-loop, LLM-driven simulation framework for human-centric embodied AI that extends Vision-and-Language Navigation (VLN) to support active direction inquiry—agents proactively ask human avatars for help mid-navigation. It ships a large-scale benchmark dataset (63,429 frames, 17+ hours) and demonstrates that enabling interaction doubles human navigation success rates while exposing a large gap in current VLN model capabilities.
Problem¶
Existing VLN systems rely on static, one-shot instructions with no mechanism for mid-episode clarification; simulators lack dynamic human agents and socially grounded interaction; and high-level social intention modeling is decoupled from navigation planning. Together, these prevent evaluation of agents that must assess uncertainty, seek information, and adapt in real time.
Method¶
FreeAskWorld is built on Unity and integrates: - LLM-driven people simulation: character profiles (age, culture, occupation, personality) → daily schedules → navigation style labels (landmark use, direction type, distance description, utterance length) → human-like instruction generation via LLM. - Hierarchical planning for avatars: high-level LLM activity planner → finite state machine for subtask selection → A* + Social Force Model for local navigation. - Direction Inquiry Task: the navigating agent can issue inquiry actions at any point to request fresh directions from nearby avatars; each inquiry increments NDI. - Data generation pipeline: scene randomization (weather, time of day), panoramic RGB + six 90° perspective cameras at 1 Hz, 2D occupancy heatmaps, dialog histories, bounding boxes, segmentation masks. - Closed-loop evaluation: WebSocket-based synchronous framework linking server-side models to the Unity simulator via NAT traversal, 1 Hz control loop.
Key Contributions¶
- Interactive closed-loop simulator with LLM-driven dynamic human avatars, traffic, and weather subsystems.
- Direction Inquiry Task benchmark: a novel VLN extension evaluating self-assessment, information-seeking, and real-time replanning.
- FreeAskWorld Dataset: 63,429 annotated frames, 17+ hours of interaction data, 16 object categories, six data modalities (visual, geometric, panoramic, interaction, spatial, metadata), continuous in/outdoor environments.
- Baselines and fine-tuned variants of ETPNav and BEVBert evaluated under both open-loop (L2 error) and closed-loop (SR, SPL, NE, OSR, NDI) protocols.
- Demonstration that interaction is an independent information modality: human SR doubles (40.2% → 82.6%) when direction inquiry is enabled.
Results¶
- Open-loop: ETPNav-FT and BEVBert-FT achieve ~50% reduction in L2 trajectory error vs. base models; BEVBert-FT achieves best open-loop performance.
- Closed-loop (Table 2):
| Method | SR↑ | SPL↑ | NE↓ | OSR↑ | NDI |
|---|---|---|---|---|---|
| Human (no ask) | 40.2% | 38.2 | 18.3 m | 41.3% | 0.0 |
| Human (ask) | 82.6% | 71.2 | 3.49 m | 82.6% | 0.78 |
| ETPNav | 0.0% | 0.0 | 32.9 m | 0.0% | 0.0 |
| BEVBert | 0.0% | 0.0 | 31.0 m | 0.0% | 0.0 |
| ETPNav-FT | 0.0% | 0.0 | 31.6 m | 1.1% | 0.0 |
| BEVBert-FT | 0.0% | 0.0 | 30.0 m | 0.0% | 0.0 |
- Fine-tuned models improve NE and OSR but achieve zero SR; none issue any direction inquiries (NDI = 0).
Limitations¶
- All current VLN baselines achieve 0% SR in closed-loop; models never autonomously trigger inquiry actions, leaving the core novel capability entirely untested by existing architectures.
- Dataset is fully synthetic (Unity-generated); sim-to-real transfer is unvalidated.
- Human baseline is extremely small (four participants), limiting statistical reliability of the human upper bound.
- Current benchmark focuses exclusively on direction inquiry; more complex social tasks (negotiation, coordination, trust-building) are deferred to future work.
- Models run at only 1 Hz with an RTX 3080/3060 split; latency constraints may disadvantage reactive models.
Relevance to Agentic AI / LLM Agents¶
FreeAskWorld directly operationalizes a core agentic capability: uncertainty-driven information seeking—deciding when and how to query the environment for missing information rather than acting on incomplete priors. The Direction Inquiry Task is a concrete benchmark for evaluating proactive information gathering in embodied settings, which is tightly coupled to agent self-assessment and replanning loops. The LLM-driven avatar pipeline (profile → schedule → contextually grounded natural language) is also relevant to multi-agent simulation, where realistic human co-agents are needed to train or evaluate interactive policies. The finding that interaction itself is an information modality (doubling human SR) provides empirical grounding for architectures that treat dialogue as perception.