Skip to content

USTBench: Benchmarking and Dissecting Spatiotemporal Reasoning Capabilities of LLMs as Urban Agents

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; USTBench: spatiotemporal reasoning capabilities of LLMs as urban agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

USTBench is the first benchmark to evaluate LLMs as urban agents across four decomposed spatiotemporal reasoning dimensions—understanding, forecasting, planning, and reflection—using 62,466 structured QA pairs across nine real-world urban tasks. Evaluating 14 leading LLMs reveals that while LLMs handle understanding and forecasting reasonably well, they systematically fail at long-horizon planning and reflective adaptation. Notably, reasoning models (e.g., DeepSeek-R1) trained on general logical/mathematical tasks do not consistently outperform non-reasoning LLMs on urban spatiotemporal tasks.

Problem

Existing urban LLM benchmarks (STBench, CityBench, CityGPT) suffer two critical gaps: (1) they rely solely on outcome-based metrics (e.g., prediction accuracy, traffic efficiency), masking intermediate reasoning deficits—a model may arrive at a correct answer via flawed reasoning that will fail under distribution shift; and (2) they neglect reflective reasoning, failing to assess whether agents can diagnose past errors from environmental feedback and adapt in dynamic urban contexts.

Method

The authors build UAgentEnv, an interactive city environment supporting nine urban tasks (four prediction: next-POI, congestion forecasting, socio-economic indicator prediction, traffic OD; five decision-making: traffic signal control, POI placement, route planning, road planning, urban planning). The formal agent definition is \(E = \langle S, A, O, T \rangle\) with the agent operating at each step \(i\) to either execute action \(a_i \in A\) or predict future states \(\{s_{i+1}, \ldots, s_{i+\Delta}\}\).

Spatiotemporal reasoning is decomposed into four evaluable stages, each assessed via QA pairs: - Understanding (27,000 QAs): eight sub-skills covering spatial (distance, adjacency, connectivity) and temporal (duration, chronology, local extrema, periodicity, trend) inference. - Forecasting (15,336 QAs): next-step prediction from historical observations. - Planning (15,000 QAs): action selection grounded in simulation-based exhaustive search ground truth: \(a^*_i = \arg\max_{a_i \in A} \sum_{j=0}^{H} \max_{a_{i+1},\ldots} \mathbb{E}[\gamma^j R(a_{i+j}) \mid a_i]\). - Reflection (8,130 QAs): given prior action/prediction, current observation, and feedback \(f_i\), assess correctness and revise.

Observations are verbalized as sparse adjacency matrices (spatial) and discrete time-series attribute sequences (temporal). Decision-making QA scenarios are generated via a semi-stochastic heuristic policy \(\pi_g(o)\) with exploration coefficient \(\epsilon\) to ensure diversity.

Key Contributions

  • First benchmark explicitly decomposing urban LLM agent reasoning into four process stages rather than relying on end-to-end outcomes.
  • UAgentEnv: an interactive urban environment with standardized inputs/outputs across nine heterogeneous real-world tasks, enabling both QA collection and end-to-end evaluation.
  • 62,466 structured QA pairs (40% understanding, 60% higher-order reasoning).
  • Evaluation of 14 LLMs spanning non-reasoning (Qwen2.5, GLM4, Llama3.3-70B, GPT-4o) and reasoning models (DeepSeek-R1 family, QwQ-32B, GLM-Z1, gpt-oss-20B, o4-mini), including same-architecture paired comparisons (e.g., Qwen2.5-32B vs. QwQ-32B).
  • Diagnostic ablations revealing the dependency structure: planning is a higher-order ability that builds on understanding and forecasting.

Results

  • Spatiotemporal understanding (Table 3): o4-mini achieves the best overall score (0.7924); GPT-4o leads non-reasoning models (0.7259). DeepSeek-R1-7B collapses on spatial tasks (adjacency: 0.045, connectivity: 0.034, overall: 0.2936—near random baseline of 0.25). Trend analysis is universally weakest sub-skill (top model: o4-mini at 0.2340 on trend, vs. 0.993 on duration).
  • Reasoning vs. non-reasoning gap: DeepSeek-R1 full model slightly underperforms Llama3.3-70B on congestion prediction outcome-based metrics; process-based analysis traces this to temporal trend understanding failures, a deficiency invisible from outcome metrics alone.
  • Planning and reflection: LLMs broadly struggle; long-horizon planning is identified as the most difficult capability; reflection remains a significant weakness even for leading reasoning models.
  • General reasoning does not transfer: Reasoning models trained on math/logic tasks do not consistently outperform non-reasoning counterparts on domain-specific urban spatiotemporal tasks, indicating a generalization gap.

Limitations

  • Paper text is truncated before full quantitative results for forecasting, planning, and reflection QA, and before ablation study data; reported numbers are thus partial.
  • Planning ground truth requires simulation-based exhaustive search over horizon \(H\), which approximates but may not perfectly reflect real-world optimal policies under full stochasticity.
  • Benchmark is confined to structured, verbalized spatiotemporal inputs; multimodal or map-image inputs are not considered.
  • The heuristic policy for generating decision-making QA scenarios (Equation 1) may not cover all realistic decision trajectories, limiting scenario diversity.
  • Domain adaptation techniques to close the identified reasoning gaps are not explored within the paper.

Relevance to Agentic AI / LLM Agents

USTBench directly probes the agentic loop—perceive → forecast → plan → reflect—in a dynamic, closed-loop environment, making it a rigorous diagnostic tool for LLM agent research beyond static QA. The finding that general-purpose reasoning ability (e.g., chain-of-thought, RLHF on math) does not transfer to domain-specific agentic tasks is a strong signal for the community: capability on generic benchmarks is insufficient for real-world agent deployment. The reflection dimension is particularly novel, targeting a capability (self-correction from environmental feedback) that is central to long-horizon autonomous agents but rarely benchmarked at a process level. This work complements agent benchmarks like AgentBench and τ-bench by shifting from task-completion rates to mechanistic diagnosis of where in the reasoning chain agents break down.