Skip to content

DiMA: An LLM-Powered Ride-Hailing Assistant at DiDi

🕒 Published (v1): 2025-02-12 10:33 UTC · Source: Arxiv · Venue: KDD 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DiMA is a production LLM assistant for DiDi's ride-hailing app that orchestrates spatiotemporal tool-calling, cost-tiered LLM routing, and continual self-improvement via real+synthetic dialogue. Deployed since May 2024, it achieves 93% order planning accuracy and 92% response quality in production. Offline experiments show up to 70.23% / 321.27% gains over SOTA agent frameworks with 0.72×–5.47× lower latency.

Problem

General LLMs (including GPT-4o and Qwen2.5-72B) fail on production ride-hailing tasks: <80% accuracy on spatial/temporal intent extraction, no mechanism to handle multi-turn progressive order planning, and uniform LLM usage is too slow and expensive for strict online latency requirements.

Method

DiMA consists of three tightly coupled modules:

Spatiotemporal-aware Tool-augmented Order Planning — A four-stage structured pipeline (tool collection+annotation → task decomposition → dynamic tool selection → structured function calls) issues API calls (e.g., POI_search(), Get_departure_time(), Route_planning_API()) to resolve ambiguous spatial/temporal trip attributes before invoking Order_create().

Cost-effective Dialog System — Three specialized repliers handle distinct goals: (1) Specialized replier for formulaic order-guidance responses, (2) Error-handling replier for infeasible requests, (3) Knowledge-enhanced replier that retrieves from a live knowledge base for policy/chit-chat queries. A cost-aware LLM configuration pre-profiles each replier's quality–latency tradeoff and statically assigns model size from a pool, routing simple policies to smaller models and complex ones to larger models.

Continual Fine-tuning — Real-world multi-turn conversations are stored periodically in MySQL. A role-playing simulator (GPT-4o as both passenger and assistant) generates synthetic dialogues seeded from historical trip orders and synthetically generated user profiles. A GPT-based quality filter removes low-quality samples. The backbone LLM is periodically fine-tuned on this blended dataset to handle cold-start and distributional drift.

Key Contributions

  • First LLM-based assistant deployed inside DiDi's mobile application, handling pre-ride, in-ride, and post-ride stages via a single conversational interface.
  • Spatiotemporal tool-augmented planning pipeline that maps ambiguous natural-language trip intent to structured POI coordinates and exact datetimes via external API calls.
  • Cost-aware LLM routing that per-replier selects model size, achieving latency reduction without accuracy loss.
  • Continual fine-tuning schema combining real-world interactions with role-playing-simulated dialogues and GPT-based quality filtering.
  • Release of code and an MCP service (mcp.didichuxing.com/api) exposing DiDi's ride-hailing tools to the research community.

Results

  • Online (2 months production): 93% order planning accuracy; 92% response generation accuracy.
  • Offline vs. 3 SOTA agent frameworks (Beijing + Shanghai synthetic datasets): up to +70.23% on order planning; up to +321.27% on response generation.
  • Latency vs. large LLMs (Qwen2.5-72B, Mistral-123B, Llama3.1-405B): 0.72×–5.47× reduction while maintaining comparable quality.
  • Baseline LLM ceiling: GPT-4o and Qwen2.5-72B achieve <80% on spatiotemporal intent extraction without DiMA's tooling.

Limitations

  • Offline evaluation relies entirely on synthetic datasets for Beijing and Shanghai; generalization to other cities or languages is not evaluated.
  • The continual fine-tuning cadence and its effect on catastrophic forgetting are not quantified.
  • Cost-aware LLM configuration is determined by static profiling; no online adaptive routing is described.
  • Role-playing simulation uses synthetically generated (not real) user profiles, which may not capture true behavioral diversity.
  • The paper is truncated before the full deployment and ablation sections.

Relevance to Harnesses / Meta-Harnesses

DiMA is a concrete production example of a multi-tier LLM dispatch harness: a meta-controller selects which model (and which replier) handles each sub-task at inference time based on pre-profiled cost-quality curves — the canonical harness pattern of routing tasks to the cheapest capable model. The continual fine-tuning loop, seeded by both real interactions and a GPT-4o-driven role-playing simulator with automated quality filtering, is a self-improving harness that closes the loop between deployment and training. The four-stage tool-selection pipeline (annotate → decompose → select → call) is a reusable template for building domain-specific tool-augmented agents, directly applicable to harness design for other structured-output domains.