Skip to content

CitySeeker: How Do VLMs Explore Embodied Urban Navigation with Implicit Human Needs?

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; VLMs interpreting implicit human needs in urban embodied navigation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CitySeeker is a benchmark of 6,440 real-world urban navigation trajectories across 8 cities that tests VLMs on implicit human need resolution (e.g., "I'm thirsty") rather than explicit step-by-step instructions. Evaluating 27 VLMs reveals uniformly low performance—the best model, Qwen2.5-VL-32B-Instruct, achieves only 21.1% TC-Proximity—exposing critical gaps in long-horizon spatial reasoning, experiential recall, and commonsense inference. The paper further proposes BCR (Backtracking, Spatial Cognition, Memory-Based Retrieval), a suite of human-cognitive-mapping-inspired strategies that yield modest but consistent gains.

Problem

Existing Vision-Language Navigation (VLN) benchmarks focus on explicit step-by-step instructions and predominantly indoor or constrained outdoor settings. Real-world urban navigation requires agents to interpret implicit needs—functional ("I'm thirsty" → infer viable POI categories), spatial (a Starbucks may be inside a nearby mall), and semantic (a "romantic" restaurant)—without predefined navigation directives. No large-scale outdoor benchmark assessed this implicit-need-driven navigation across globally diverse cities until CitySeeker.

Method

Benchmark Construction. Street-level panoramas from Google and Baidu Street View (post-2024) are sampled every 20 meters across 8 cities (Beijing, Shanghai, Shenzhen, Chengdu, Hong Kong, London, New York, plus one implicit from the dataset) and stored as a navigable graph in Neo4j, with edges annotated by azimuth and distance. POIs within a 50 m radius of each node are linked via (Node) –has–> (VisiblePOI) triplets. Routes of 5–25 steps are generated via A* between a start node and a goal node containing the target POI, producing 6,440 trajectories across 7 implicit-need categories (Basic POI, Brand-Specific, Transit Hub, Latent POI, Abstract Demand, Inclusive Infrastructure, Semantic Preference). A 1,257-instance subset serves as the test set.

Evaluation Framework. At each step \(t\), the agent receives natural language instruction \(W\) and observation set \(O_t = \{o_{t,1}, \ldots, o_{t,n}\}\) (perspective-cropped views from a panorama). A ReAct-style policy \(\pi_\Theta\) generates \((Φ_t, a_t, c_t)\)—rationale, chosen view index, and confidence—iterating up to 35 steps. Evaluation metrics: TC-Exact (TCE, strict endpoint match), TC-Proximity (TCP, ≤50 m geodesic threshold), TC-Category (TCC), SPL, nDTW, and Shortest-Path Distance (SPD). Each step is stateless (no persistent memory) to isolate core spatial reasoning.

BCR Exploratory Strategies. Three strategy families are analyzed: (B) Backtracking—variants enabling self-correction upon detected error; (C) Spatial Cognition Enrichment—augmenting the agent with structured map or spatial context; (R) Memory-Based Retrieval—providing the agent access to past observations. Experiments run on a 650-sample mini-subset.

Key Contributions

  • First large-scale benchmark for implicit-need-driven embodied urban navigation across 8 globally diverse cities with 6,440 trajectories and 7 goal-category types.
  • Systematic zero-shot evaluation of 27 VLMs spanning proprietary (GPT-4o, Gemini-2.5-Pro, o4-mini) and open-source (Qwen2.5-VL, InternVL3, Llama-4, etc.) models.
  • Identification of three core VLM bottlenecks: error accumulation in long-horizon reasoning, inadequate spatial cognition, and deficient experiential recall.
  • BCR framework of cognitive-mapping-inspired strategies (Backtracking, Spatial Cognition Enrichment, Memory-Based Retrieval) providing ablatable analysis of each bottleneck.
  • Human baseline and cross-lingual experiments quantifying the human–VLM gap and ruling out language as the primary performance driver in cross-city variation.

Results

  • Best VLM (Qwen2.5-VL-32B-Instruct): TCP = 21.1%, TCE = 2.6%, SPL = 12.7%; human baseline TCP = 30.1%, SPL = 21.2%.
  • Several models (e.g., Llava-Qwen2-7B, TCP = 6.9%) fall below the Random Choice baseline (TCP = 13.9%).
  • GPT-4o TCP = 18.3%; Gemini-2.5-Pro TCP = 17.3%; InternVL3-38B TCP = 19.3%.
  • Hardest category: Transit Hub (best VLM TCP = 19.6% for Qwen2.5-VL-32B; humans = 34.9%); easiest: Brand-Specific (best = 30.4% for Qwen2.5-VL-32B).
  • Performance degrades sharply beyond ~20 navigation steps; nDTW becomes highly scattered at 35 steps.
  • BCR best results (TCP on mini-subset): R1 strategy on Qwen2.5-VL-32B lifts TCP from 19.9% → 26.9%; B3 raises GPT-4o-Mini from 12.5% → 18.2%.
  • Providing a global 2D map degraded task completion (ablation), indicating difficulty fusing map and visual modalities.
  • City performance gap: GPT-4o variants score highest in New York, poorly in Beijing; cross-lingual experiments show language is not the primary driver.

Limitations

  • Best TCP of 21.1% is far below human performance (30.1%), and TCE remains below 3% for all models, indicating the task is largely unsolved.
  • BCR experiments run on a 650-sample mini-subset only, limiting statistical robustness of strategy conclusions.
  • Panoramic data is from Google/Baidu Street View (≥2024) and may not reflect fully dynamic real-time conditions; temporal drift of POIs is not addressed.
  • Route validation and Need-to-POI mappings required manual verification; designer bias partially mitigated by a cross-cultural survey (\(N=120\), 83.39% agreement) but not eliminated.
  • The stateless per-step evaluation isolates spatial reasoning but does not reflect deployable agent architectures with persistent memory.
  • Coverage limited to 8 metropolitan areas with uneven geographic distribution (6 Chinese cities vs. 2 Western cities).

Relevance to Vision-Language Models

CitySeeker directly stresses a capability gap that most VLN benchmarks ignore: translating abstract, implicit natural language needs into sequential visual grounding decisions in open-world environments, exactly the setting where current VLMs are deployed as AR assistants, service robots, and drone controllers. The benchmark's finding that even Qwen2.5-VL-32B tops out at 21.1% TCP—and that adding map information hurts—reveals that multi-modal fusion of symbolic maps with visual streams is still fundamentally broken. The BCR framework connects to broader VLM research on spatial mental models and retrieval-augmented agents, providing a concrete evaluation harness for measuring progress on long-horizon embodied reasoning in VLMs.