Skip to content

VLMLight: Safety-Critical Traffic Signal Control via Vision-Language Meta-Control and Dual-Branch Reasoning Architecture

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VLMLight is a traffic signal control (TSC) framework that combines a vision-language model (VLM) for scene understanding with a dual-branch architecture: a fast RL policy for routine traffic and a multi-agent LLM deliberation chain for safety-critical events such as emergency vehicle passage. A meta-controller LLM routes each decision to the appropriate branch in real time. On three real-world intersections, VLMLight cuts emergency vehicle waiting time by up to 65% versus RL-only baselines while degrading routine throughput by less than 1%.

Problem

Existing TSC methods face a three-way gap: rule-based methods are rigid; RL-based methods use vectorized, semantically impoverished state representations that cannot detect special vehicles or reason about safety priorities; LLM-based methods rely on hand-crafted text descriptions (no visual grounding) and are too slow for real-time control. No prior simulator supports multi-view image input at intersections, blocking vision-language integration.

Method

VLMLight operates in four stages:

  1. Scene Understanding: A VLM (Qwen2.5-VL-32B) processes multi-view camera images {I₁…I_D} from a custom SUMO-integrated simulator—with bird's-eye and per-approach directional views—and generates per-direction natural-language descriptions {T₁…T_D} including lane semantics, congestion level, and emergency vehicle presence.

  2. Safety-Prioritized Meta-Control: An LLM meta-controller (AgentModeSelector) reads {T₁…T_D} and routes to either the fast RL branch (no critical conditions) or the deliberative reasoning branch (emergency vehicle detected, conflicting priorities, etc.).

  3. Routine Control: A PPO-trained Transformer encodes spatio-temporal features over 12 movements × 5 time steps and selects a signal phase with low latency.

  4. Deliberative Reasoning: Three sequential LLM agents (Qwen2.5-72B) collaborate: AgentPhase maps directional descriptions to phase-level summaries aligned with the discrete action space; AgentPlan selects the optimal phase with a textual rationale; AgentCheck verifies feasibility against the legal phase set and falls back to the closest valid alternative if needed. The final decision is emitted as JSON.

Key Contributions

  • First image-based TSC simulator with configurable multi-view rendering built on SUMO, enabling visual perception at intersections.
  • VLMLight framework: LLM meta-controller that dynamically selects between a pre-trained RL policy and a three-agent deliberative reasoning chain based on real-time VLM scene summaries.
  • Demonstrated 60–65% reduction in emergency vehicle waiting time over RL-only baselines with <1% degradation in routine ATT/AWT, across three geographically diverse real-world intersection layouts.
  • End-to-end inference latency of 11.48 s for the full deliberative branch, fitting within a typical 13 s signal phase buffer.

Results

  • Songdo (KR): AEWT drops from 22.04 s (best RL, UniTSA) to 7.48 s (VLMLight), a 66% reduction; ATT increases marginally from 86.80 s to 87.14 s (<1%).
  • Yau Ma Tei (HK): AEWT drops from 5.17 s to 2.17 s (58% reduction); ATT increases from 38.07 s to 39.80 s.
  • Massy (FR): AEWT drops from 7.35 s (A-CATs, best RL) to 2.60 s (65% reduction); ATT increases from 57.73 s to 60.84 s.
  • Vanilla-VLM (scene descriptions without structured dual-branch) shows high variance and frequent action errors in complex intersections (e.g., Yau Ma Tei ATT 62.45 ± 8.76 s vs. VLMLight 39.80 ± 1.65 s).
  • Ablation: removing AgentPhase raises AEWT from 7.48 s to 9.03 s; removing AgentCheck has minimal effect (7.48 → 8.12 s without it); adding AgentCheck reduces latency from 10.97 s to 11.48 s (small overhead, large reliability gain).

Limitations

  • The custom simulator lacks diverse weather and lighting conditions, so visual robustness under rain, night, or glare is unvalidated.
  • All experiments are single-intersection; multi-intersection coordination with distributed VLM/LLM reasoning at scale is not studied.
  • LLM inference cost (Qwen2.5-72B) is not analyzed in terms of compute or deployment economics.
  • Evaluation relies on a single VLM backbone (Qwen2.5-VL-32B); generalization to other VLMs is only discussed in appendix.

Relevance to Vision-Language Models

VLMLight is a direct application of VLMs as grounded perceptual front-ends: raw intersection images are converted to structured language descriptions by a VLM, which then serve as the interface between visual observation and LLM-based reasoning—a paradigm increasingly common in embodied and agentic VLM research. The paper demonstrates that VLMs enable detection of semantically rich but visually implicit cues (e.g., vehicle type, emergency status) that purely vectorized RL states cannot capture. The modular multi-agent LLM reasoning chain (Phase Reasoning → Signal Planning → Rule Verification) is an instance of tool-augmented, role-specialized LLM collaboration, relevant to ongoing work on VLM-based agents for real-world control tasks. The paper also highlights the latency-accuracy tradeoff inherent in deploying VLMs in real-time closed-loop control, a fundamental challenge for the VLM-agent community.