RAST-MoE-RL: A Regime-Aware Spatio-Temporal MoE Framework for Deep Reinforcement Learning in Ride-Hailing¶

🕒 Published (v1): 2025-12-13 20:49 UTC · Source: Arxiv · Venue: ICML 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

RAST-MoE-RL applies a Mixture-of-Experts (MoE) encoder within a PPO-based RL framework to the adaptive delayed matching problem in ride-hailing, where an agent decides per-zone whether to immediately assign drivers to passengers or delay. A 12M-parameter regime-aware spatio-temporal MoE encoder, combined with a physics-informed travel-time surrogate and an anti-reward-hacking penalty, outperforms dense baselines on real Uber data from San Francisco.

Problem¶

Existing RL methods for ride-hailing matching use shallow or monolithic encoders that cannot capture non-stationary supply–demand regimes (morning peaks vs. off-peak) or time-varying congestion. They also approximate travel times with static cost matrices and are prone to reward hacking (e.g., always matching immediately or holding indefinitely), limiting policy transferability across heterogeneous demand conditions.

Method¶

The framework has three interlocking components:

RAST-MDP formulation: Adaptive delayed matching is cast as a regime-aware MDP with zone-wise binary actions (match-now vs. hold), a spatio-temporal state vector encoding unmatched passengers/drivers and stochastic arrival rates across N zones, and a physics-informed travel-time surrogate built on Macroscopic Fundamental Diagrams (MFDs). Zone–hour speeds are derived from MFD-consistent flow/density aggregates; O(1) travel-time queries use precomputed hourly OD tables via static shortest paths.
Anti-hacking reward: Reward combines incremental matching and pickup delay costs with an online Lagrange multiplier λ that adaptively penalizes service-quality violations (fraction of late matches/pickups exceeding a 5% tolerance), preventing collapse to pathological hold-all or match-immediately strategies.
RAST-MoE encoder: A lightweight self-attention Transformer tokenizes per-zone supply/demand features with sinusoidal and learnable positional embeddings, global average-pools to a state vector h, then routes h through a sparse top-K MoE layer (two-layer MLP gating with GELU). Only K of E experts activate per step; outputs are softmax-weighted and aggregated. Load balancing uses a lightweight mechanism tolerating imbalanced utilization to permit genuine specialization. The encoder replaces the shared feature extractor in PPO; actor and critic heads remain linear. The framework is also compatible with A2C, ACER, and GRPO.

Key Contributions¶

Formalization of adaptive delayed matching as RAST-MDP with explicit regime-heterogeneous state, combinatorial zone-wise actions, and a physics-informed MFD-based travel-time surrogate scalable to millions of RL rollouts.
Anti-hacking reward design using an online adaptive multiplier that enforces service-quality constraints without fixed coefficient tuning.
RAST-MoE encoder (12M params) with global-state routing and imbalance-tolerant load balancing, demonstrating that purposeful uneven expert specialization outperforms uniform balancing in cyclic demand settings.
Empirical demonstration that both frequent and rare experts encode distinct, indispensable demand regimes—masking either group degrades performance.

Results¶

Evaluated on 2019 CPUC TNC data (San Francisco, ~170k trips/day); training and test sets are temporally disjoint.

RAST-MoE (16 experts, top-4, PPO) vs. PPO baseline: +13% total reward, −10% matching wait, −15% pickup wait.
Best configuration (E=16, K=4, 12M params) outperforms parameter-matched dense Transformers (12M and 48M), plain MLPs, ResMLP, A2C, ACER, and DQN on both matching and pickup waits.
Adaptive reward vs. fixed 8:1 ratio: fixed ratio drives pickup waits above 1,582±225 s (hacking); adaptive achieves 195±40 s matching wait and 567±44 s pickup wait with lowest variance.
Masking high-frequency experts or low-frequency experts in the 16-expert model both cause sharp reward drops and wait stagnation.
GRPO variant with MoE achieves comparable gains to PPO+MoE, confirming encoder compatibility across RL paradigms.

Limitations¶

Evaluated on a single city (San Francisco 2019); generalization to cities with qualitatively different street networks or demand structures is untested.
The MFD surrogate aggregates at zone–hour granularity, smoothing microscopic traffic dynamics that could matter for fine-grained decisions.
Static pre-computed route sets may not capture real-time rerouting under incidents or extreme congestion.
The combinatorial action space ({0,1}^N) is handled via factorized Bernoulli heads, which achieves global coordination only through shared state pooling—not explicit inter-zone coordination mechanisms.
Labor/equity impacts on gig-economy drivers from algorithmic matching optimization are acknowledged but not quantitatively studied.

Relevance to Agentic AI / LLM Agents¶

This work is directly relevant as a concrete case study of MoE architectures—originally developed for scaling LLMs—being adapted as modular, regime-specialized policy networks in a real-world sequential decision-making agent. The sparse-routing MoE design parallels the expert-selection mechanisms in models like DeepSeek-V3, and the paper explicitly draws on that lineage while highlighting domain-specific adaptations (imbalanced expert utilization as a feature, not a bug). The anti-reward-hacking reward design addresses a core challenge in deploying RL agents in service systems—reward misalignment—offering a practical template for agentic systems that must satisfy long-horizon service-level constraints without hand-tuned coefficients. The GRPO adaptation from LLM preference optimization to RL control further illustrates cross-pollination between LLM training techniques and agentic RL.