LocalSearchBench: Benchmarking Agentic Search in Real-World Local Life Services¶

🕒 Published (v1): 2025-12-08 11:12 UTC · Source: Arxiv · Venue: KDD 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

LocalSearchBench is the first benchmark for evaluating agentic search systems in the local life services domain (dining, accommodation, healthcare, etc.), covering 1.35M merchant profiles across 9 Chinese cities and 900 multi-hop QA tasks derived from real user queries. Even the best evaluated model (DeepSeek-V3.2) achieves only 35.60% correctness, exposing severe gaps in completeness and faithfulness for domain-specific agentic retrieval.

Problem¶

Existing agentic search benchmarks target general information retrieval and lack coverage of vertical domains with geographic constraints, multi-service coordination, and ambiguous real-world queries. Prior local-life-service work (Lan et al.) addresses only basic retrieval, leaving complex compositional and multi-hop scenarios unevaluated. This absence of domain-specific benchmarks blocks progress on training and assessing agents for the local services market.

Method¶

The benchmark is built in three interconnected layers:

Merchant database: 1.6M+ raw records from a live local-services platform are expanded from 12 to 29 fields via a Data Augmentation Agent, anonymized via a Privacy Rewriting Agent (7 sensitive fields rewritten), and validated by a Data Validation Agent (LLM-as-judge) across weighted quality dimensions—yielding 1,354,185 final profiles across 6 categories and 9 cities.
LocalRAG retrieval tool: A two-stage pipeline using Qwen3-Embedding-8B for dense vector retrieval (top-100) over city-partitioned indices, followed by Qwen3-Reranker-8B for reranking to top-20. Evaluated on 27,360 hop-level queries with NDCG/MRR metrics.
Multi-hop QA construction: 1,200 real user single-hop queries (90% from platform logs, 10% expert-enriched) are manually instantiated with city-specific context and extended to 2–4 reasoning segments (cross-merchant comparison, schedule planning, chained events, constraint resolution), targeting intelligence levels L3 (Intelligent, 66.7%) and L4 (Expert, 33.3%), with 3–5 hops per question. Reference answers are generated via iterative LocalRAG + web search + Claude Opus 4.5, then validated hop-by-hop and by three domain experts.

LocalPlayground wraps evaluation: a Search Agent (ReAct loop, up to N=5 rounds, using LocalRAG and Baidu web search) and a Validation Agent (Claude Opus 4.5 as LLM-judge) scoring 7 answer-quality dimensions and 4 trajectory-quality dimensions.

Key Contributions¶

First comprehensive offline benchmark for agentic search in real-world local life services, with 1.35M merchant profiles and 900 multi-hop QA pairs.
Systematic data pipeline: augmentation (+17 fields), privacy rewriting, and LLM-as-judge validation preserving retrieval signal (16.5% NDCG@10 gain from augmentation; only 0.34–0.40% degradation from anonymization).
LocalRAG two-stage retrieval achieving NDCG@10 = 0.84, MRR@10 = 0.82.
LocalPlayground evaluation framework with 11 metrics (7 answer + 4 trajectory) including faithfulness, causal coherence, and search efficiency.
Systematic evaluation of 16 LRMs (thinking and non-thinking), revealing that top performance caps at 35.60% correctness.

Results¶

Best overall correctness: DeepSeek-V3.2 (w/ thinking) 35.60%; GLM-4.6 32.83%; Gemini-2.5-Pro 32.34%; o3(high) 31.53%.
Worst correctness: GPT-4.1 18.47%, Qwen3-32B (w/o thinking) 19.76%.
Thinking vs. non-thinking: correctness 30.49% vs. 21.30%; completeness 64.59% vs. 49.10% on average.
Completeness (average across all models): 60.32%; Faithfulness (average): 30.72%—both indicate systematic gaps.
L3 vs. L4 difficulty: correctness drops from 30.57% (L3) to 25.10% (L4) across all 16 models.
Tool ablation (DeepSeek-V3.2): no tools → 9.78%; +LocalRAG → 27.00%; +LocalRAG & web search → 35.60%.
Max round sensitivity: N=5 optimal (35.60%); performance degrades for N>5, and faithfulness declines monotonically with more rounds.
LocalRAG quality: κ̄ = 0.74 (88.34% human-machine agreement); overall augmented data quality score 0.86, privacy-rewriting quality 0.92.

Limitations¶

Benchmark is China-specific (9 Chinese cities, Baidu APIs, Mandarin queries); generalizability to other locales is unstated.
L5 (AGI-level, cross-platform dynamic tasks) are excluded because they require live cross-platform interactions that cannot be simulated offline.
Temporal elements are anchored to static timestamps, reducing evaluation of time-sensitive or real-time queries.
LLM-as-judge (Claude Opus 4.5) is used for both answer generation and validation, introducing potential self-consistency bias (mitigated by using a distinct judge model from evaluated models, but the judge itself generated the reference answers).
Only 900 QA pairs despite the large merchant database; distribution balance is maintained by prioritizing underrepresented categories.
No fine-tuned domain-specific model is trained or evaluated; the benchmark highlights the gap but provides no remedy.

Relevance to Agentic AI / LLM Agents¶

LocalSearchBench directly stress-tests the multi-step tool-use and retrieval-augmented reasoning pipeline that defines practical LLM agents—specifically the ReAct-style iterative query decomposition, tool orchestration (RAG + web search), and evidence synthesis loop central to production agentic search. The benchmark reveals that even frontier thinking models fail to break 36% correctness on real-world compositional queries, quantifying a concrete capability ceiling for agents operating in structured-but-ambiguous vertical domains. The evaluation framework's trajectory metrics (action relevance, causal coherence, search efficiency) are directly applicable to debugging and improving general-purpose agent architectures. This work also contributes a reproducible evaluation harness and large-scale domain-specific database that practitioners building location-aware or service-domain agents can use to measure progress.