Skip to content

VLMLight: Safety-Critical Traffic Signal Control via Vision-Language Meta-Control and Dual-Branch Reasoning Architecture

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLMLight is a traffic signal control (TSC) framework that uses an LLM as a meta-controller to dynamically route between a fast RL policy (routine traffic) and a multi-agent LLM deliberative reasoning pipeline (safety-critical events like emergency vehicles). It introduces the first vision-enabled TSC simulator with multi-view image inputs, enabling VLM-grounded scene understanding. The dual-branch architecture reduces emergency vehicle waiting time by up to 65% over RL-only baselines while incurring less than 1% degradation in routine throughput.

Problem

Existing TSC methods face a trilemma: rule-based systems lack adaptability; RL agents use simplified vectorized states that omit visual/semantic cues needed for safety-critical decisions (e.g., ambulance priority); and LLM-only approaches are too slow for real-time control and rely on hand-crafted textual state descriptions rather than perceptual grounding. No prior TSC simulator supported multi-view image rendering, blocking vision-language integration.

Method

VLMLight operates in four sequential stages:

  1. Scene Understanding: A VLM agent (Qwen2.5-VL-32B) processes multi-view intersection images (N, E, S, W cameras) and generates per-direction natural language summaries capturing lane semantics, congestion level, and special vehicle presence.
  2. Safety-Prioritized Meta-Control: An LLM agent (AgentModeSelector) reads the scene summaries and classifies the situation as routine or safety-critical, routing to the appropriate branch.
  3. Routine Control (fast branch): A PPO-trained RL policy takes a spatio-temporal feature tensor (12 movements × 5 time steps) through a Transformer encoder and selects a signal phase from a discrete action set.
  4. Deliberative Reasoning (slow branch): Three specialized LLM agents (Qwen2.5-72B) collaborate sequentially: AgentPhase maps directional descriptions to phase-level summaries; AgentPlan selects a phase with a textual rationale; AgentCheck verifies the chosen phase against feasible phase set A_t and corrects if invalid.

The simulator is built on SUMO with added multi-view rendering, evaluated on three real-world intersections (Songdo KR, Yau Ma Tei HK, Massy FR).

Key Contributions

  • First image-based TSC simulator supporting configurable multi-view visual inputs (bird's-eye + directional cameras) integrated with SUMO dynamics.
  • LLM meta-controller that routes between fast RL and deliberative LLM branches based on semantic scene content, avoiding LLM overhead in routine cases.
  • Modular multi-agent deliberative reasoning pipeline (Phase Reasoning → Signal Planning → Rule Verification) for structured, auditable safety-critical decisions.
  • Empirical demonstration of a 65% reduction in emergency vehicle waiting time with <1% routine efficiency degradation and full inference latency of 11.48 s (fits within 13 s signal buffer).

Results

  • Songdo: AEWT drops from 22.04 s (best RL, UniTSA) to 7.48 s (VLMLight); ATT increases marginally from 86.80 s to 87.14 s (<0.4%).
  • Yau Ma Tei: AEWT drops from 5.17 s (best RL) to 2.17 s (VLMLight); ATT increases from 38.07 s to 39.80 s (~4.5%).
  • Massy: AEWT drops from 7.35 s (best RL, A-CATs) to 2.60 s (VLMLight); ATT increases from 57.73 s to 60.84 s (~5%).
  • Vanilla-VLM (no meta-control, no structured agents) performs significantly worse on routine ATT and produces frequent action errors in complex intersections despite some AEWT gains.
  • Ablation (Songdo): removing AgentPhase raises AEWT from 7.48 s to 9.03 s; removing AgentCheck has minimal impact (rare invalid actions); full pipeline latency is 11.48 s vs. 7.10 s without any LLM agents.

Limitations

  • Simulator lacks diverse weather and lighting conditions, limiting evaluation of visual robustness under fog, rain, or night-time scenarios.
  • All experiments are single-intersection; multi-intersection coordination with vision-language reasoning across a network is unaddressed.
  • LLM inference latency (11.48 s) consumes nearly the full 13 s signal phase buffer, leaving little margin for longer reasoning chains or slower model deployments.
  • Emergency vehicle density and scenario diversity in evaluation is limited (one vehicle type: ambulance/fire truck; synthetic injection).

Relevance to Agentic AI / LLM Agents

VLMLight is a concrete instantiation of the hierarchical meta-controller + specialist agent pattern central to agentic AI: a routing LLM decides whether to invoke a slower, more capable multi-agent reasoning chain, directly addressing the latency-capability tradeoff that constrains LLM deployment in real-time environments. The deliberative branch's three-agent pipeline (perceive → plan → verify) mirrors tool-use and reflection loops common in LLM agent frameworks, here grounded in a physical control loop with hard latency constraints. The work is also relevant to the emerging line of VLM-grounded agents: rather than using hand-crafted text observations, the system uses a VLM to construct a structured world model that downstream LLM agents reason over. The explicit legality-checking agent (AgentCheck) demonstrates a pattern of constrained action spaces and safety guardrails that generalizes beyond TSC to any agentic system operating in rule-governed environments.