Skip to content

Beyond Itinerary Planning-A Real-World Benchmark for Multi-Turn and Tool-Using Travel Tasks

🕒 Published (v1): 2025-12-27 18:25 UTC · Source: Arxiv · Venue: ACL 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TravelBench is a real-world travel-planning benchmark built from Alibaba Amap user logs, featuring 1,100 instances across three subtasks—single-turn, multi-turn (with implicit preference elicitation), and unsolvable—each backed by a reproducible offline sandbox with ~200K cached real tool responses. Evaluation of 15+ frontier and open-source LLMs reveals that even top models score ~75/100 overall and exhibit severely imbalanced capabilities across the three subtasks.

Problem

Existing travel-planning benchmarks (TravelPlanner, ChinaTravel, TripScore) share three gaps: (1) user preferences are pre-defined or revealed by rule, not dynamically elicited through multi-turn dialogue; (2) task scope is narrowly focused on itinerary design, ignoring POI discovery, route planning, policy queries, etc.; (3) queries and preferences are synthetic or restricted to simple constraints, diverging from real-world distributions. No prior benchmark simultaneously covers real queries, multi-turn implicit preference elicitation, capability-boundary recognition (unsolvable cases), and a reproducible tool sandbox.

Method

Dataset construction: ~5,000 real queries are collected from four months of Amap logs (Aug–Nov 2025), deduplicated with K-Center-Greedy diversity selection. User preferences are de-identified via Qwen-Plus (PII removed, specific POIs abstracted to categories), then combined with LLM-synthesized demographic attributes to form user profiles visible only to a user simulator.

Subtask labeling: Solvability is determined by consensus across GPT-5.1, Qwen3-235B, and Qwen-Plus; single-turn vs. multi-turn split is determined empirically by running queries through an interactive agent framework (Gemini-3-Flash-preview + GPT-5.1, 6 runs per query, temperature 0.7) and labeling based on whether ≥4/6 runs complete without clarification questions.

Sandbox: 10 real tools (flight/train search, POI search, route planning, weather forecast, etc.) are integrated. A cache of ~200K real tool-call traces is prebuilt. Cache misses are resolved via Qwen3-Embedding-8B + Faiss top-8 retrieval followed by ICL-based LLM simulation to preserve the real-response distribution.

Scoring: Unsolvable subset uses binary accuracy on [Unsolved] tag emission. Single-turn and multi-turn use an LLM-as-a-judge (Gemini-3-Flash-preview) on 3 rubric dimensions (single) or 4 dimensions (multi, adding user_interaction), each scored 1–5, then normalized to [0,1]. A meta-judge calibrates ~3% of cases with likely over-estimated scores. A tool-use penalty multiplier (1 − error_rate) is applied per instance.

Key Contributions

  • First travel benchmark with real user queries and real tool responses (not synthetic), covering diverse task categories beyond itinerary planning across 32 provinces and 243 cities in China.
  • First benchmark to operationalize implicit preference elicitation: user profiles are withheld from the agent and must be surfaced through multi-turn dialogue.
  • Explicit unsolvable subset with three cause categories (missing tools, missing context, no executable intent), enabling evaluation of capability-boundary recognition.
  • Reproducible offline sandbox with embedding-based cache-miss simulation; offline/online score difference is ≤0.31 points.
  • Multi-component evaluation protocol: rubric-based LLM judge + meta-judge calibration + tool-call error penalty.

Results

  • Top overall score: Qwen-Plus 76.29; GPT-5.1 75.04; DeepSeek-V3.2 72.47.
  • Lightweight models (Qwen3-4B to Qwen3-235B) cluster at 53–71 overall.
  • Capability imbalance: Kimi-K2-0925 scores 94 on unsolvable but ~62 single-turn/multi-turn; DeepSeek-V3.2 scores ~83 on single/multi-turn but only 51.33 on unsolvable.
  • Thinking vs. instruction models on unsolvable: Thinking models (Kimi-K2-Th: 73.67, DeepSeek-R1: 83.67, Qwen3-235B-Th: 51.67) show no consistent advantage and sometimes underperform instruction-following variants on unsolvable detection.
  • Tool-use penalty effect: MiniMax-M2 multi-turn drops from 78.44 (raw) to 66.79 (penalized) due to 15.37% tool error rate.
  • Reasoning models show lower tool-call error rates than instruction-following counterparts, suggesting test-time compute improves tool-use reliability.
  • Benchmark stability: Judge std across 3 runs is ≤0.02; overall system std across 3 independent trials is 0.19–0.81.

Limitations

  • Sandbox tool set is limited (10 tools); real-time traffic checking and reminders are absent, forcing such queries into the unsolvable set rather than evaluating them.
  • LLM-as-a-judge scores are only directly comparable when the same judge model is used; no pairwise or trajectory-level metric is provided as an alternative.
  • Data is restricted to Chinese geography (Amap) and Chinese-language travel scenarios, limiting generalizability.
  • Single-turn/multi-turn split relies on stochastic model behavior (6 runs at temperature 0.7), introducing label noise for borderline queries.

Relevance to Agentic AI / LLM Agents

TravelBench directly probes three capabilities that define robust agentic behavior: autonomous multi-step tool use, proactive information gathering via dialogue, and knowing when to abstain—all evaluated on real-world data with a reproducible sandbox. The finding that frontier models show severe capability imbalance (e.g., strong tool use but poor unsolvability detection, or vice versa) is a concrete signal for the agent training community about which behavioral dimensions remain misaligned. The benchmark's embedding-based cache-miss simulation is a transferable infrastructure pattern for building reproducible tool-augmented agent evaluations in other domains. The explicit penalty for tool-call schema errors also advances the community's tooling for measuring agent reliability beyond task success.