Skip to content

WorkDrive: Roadwork Chain of Causation for Autonomous Driving

🕒 Published (v1): 2026-07-16 08:53 UTC · Source: Arxiv · link

Why this paper was selected

Chain-of-causation reasoning for roadwork zones; tackles hard OOD failure mode in driving VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WorkDrive addresses the failure of VLMs in roadwork zones, where temporary traffic control devices (cones, barriers, arrow boards) replace familiar cues, by constructing perception-grounded Chain-of-Causation (CoC) reasoning annotations and aligning them with trajectory prediction via GRPO. An automated perception pipeline (depth estimation + open-vocabulary detection + segmentation + tracking) extracts structured scene facts that anchor the causal annotation process, and a consistency-based RL reward enforces agreement between the model's declared lateral intent and its predicted trajectory. On the ROADWork benchmark, CoC SFT reduces ADE by 9.0% over a trajectory-only baseline, with GRPO adding a further 3.0%.

Problem

Existing VLMs for autonomous driving anchor their reasoning on familiar pre-training elements (lane markings, traffic signals, pedestrians) and default to those patterns even in work zones, where those elements are absent or overridden by temporary devices. ROADWork, the largest public work-zone dataset, provides only trajectory ground truth without reasoning annotations. The only prior VLM planner for work zones, REACT-Drive, patches outputs post-hoc via retrieval-augmented generation rather than building the model's intrinsic causal understanding of domain-specific objects. The gap is therefore twofold: (1) existing reasoning labels for driving VLMs are self-generated without external perceptual grounding; (2) SFT-based reasoning does not guarantee consistency between a model's stated rationale and its output trajectory.

Method

WorkDrive operates in three phases:

Phase I — Automated Perception Pipeline. For each video clip \(V = \{I_1, \dots, I_T\}\), four specialized models run offline: (a) Depth Anything 3 (DA3-Giant) produces affine-invariant depth maps; objects are binned into three depth levels at the 33rd and 66th per-frame percentiles. (b) Rex-Omni performs two-pass open-vocabulary detection: 45 work-zone-specific categories (ROADWork taxonomy) and 11 general traffic categories (nuScenes/Waymo); when bounding boxes overlap (IoU > 0.5), the work-zone label is retained. (c) SAM 3 generates instance masks and embeddings \((\mathbf{m}_i, \mathbf{e}_i) = \text{SAM3}(I_t, b_i)\). (d) Cross-frame tracking links instances across frames using joint spatial overlap (IoU > 0.3) and appearance similarity (cosine > 0.7). These outputs are assembled into a per-frame structured JSON.

Phase II — Chain-of-Causation Annotation. Perception outputs are rendered into a structured prompt \(P_k\) using a two-tier scheme: on-path objects (within a perspective-scaled driving corridor \(w(y) = 60 + \frac{y}{H} \times 190\) pixels) versus peripheral objects. Four causally-constrained steps follow: (1) Closed-set decision classification — the annotator LLM (Qwen3-VL-32B-Instruct or GPT-5.2high) selects longitudinal and lateral decisions from 28 predefined categories using a 6-frame window including future frames; (2) Causal factor identification — restricted to past frames only to prevent reverse-engineering from outcomes; (3) Causal chain synthesis — text-only input to produce \(R\) following the mandatory structure "[Decision] because [Key Factors]"; (4) Automated quality assessment — GPT-5.2high scores annotations on four criteria (causal coverage, correctness, proximate-cause adherence, decision minimality), retaining only samples with score \(s \geq 8\).

Phase III — SFT + Consistency RL. LoRA-based SFT trains the 8B-parameter VLM on CoC annotations formatted as multi-turn conversations with three output segments: <reason>, <meta_action> (one of five lateral intents), <trajectory> (15 pixel-coordinate waypoints). GRPO then samples \(G=4\) outputs per prompt and optimizes: $\(\mathcal{L}_\text{GRPO}(\theta) = -\mathbb{E}_{o_i \sim \pi_\theta} \left[ \frac{\exp(\beta A_i)}{\sum_j \exp(\beta A_j)} \left(\log \pi_\theta(o_i) - \lambda_\text{KL} \text{KL}[\pi_\theta \| \pi_\text{ref}]\right) \right]\)$ with advantages \(A_i = R_i - \bar{R}\) and a multiplicative reward \(R_i = R_\text{format} \times R_\text{consistency}\), where \(R_\text{format} \in \{0,1\}\) checks output structure and \(R_\text{consistency}\) measures agreement between the declared meta-action and the mean trajectory curvature direction.

Key Contributions

  • Automated multitask perception pipeline that extracts structured, verifiable scene facts from raw ROADWork frames, providing grounding absent from all existing work-zone planning datasets.
  • Four-step causally-constrained CoC annotation pipeline that redirects annotator LLM attention to work-zone-specific elements via injected perception prompts, preventing the familiar-element anchoring failure.
  • Single GRPO consistency reward (meta-action ↔ trajectory curvature) that improves planning over SFT without any explicit trajectory regression loss.
  • First validation of perception-grounded reasoning and consistency-based RL alignment in out-of-distribution work-zone scenarios.
  • Public release of code and the CoC reasoning dataset built on ROADWork.

Results

  • CoC SFT reduces trajectory ADE by 9.0% over the trajectory-only SFT baseline on ROADWork.
  • Consistency-based GRPO yields a further 3.0% ADE reduction over CoC SFT alone.
  • All general-purpose frontier VLMs and the trajectory-only SFT baseline deviate significantly from ground-truth trajectories in work-zone scenes (qualitative, Figure 1).
  • GPT-5.2high selected as Step-4 judge based on closest alignment with human ratings among three annotator candidates (quantified in §4.4, details truncated in provided text).

Limitations

  • The 9.0% and 3.0% ADE improvements are reported only against trajectory-only SFT; comparison to REACT-Drive or other direct baselines is not fully visible in the provided text.
  • Phase I perception pipeline requires running four specialized large-vision models offline, making annotation expensive and dataset-specific; inference is unaided by perception, but annotation scalability to new domains is non-trivial.
  • Only lateral consistency is used as the RL reward; longitudinal decision consistency is absent, which the paper itself notes as a minor weakness in the annotation quality examples.
  • Evaluation is restricted to a single dataset (ROADWork); generalizability to other OOD scenarios is not demonstrated.
  • CoC annotation quality depends on GPT-5.2high as sole judge (Step 4), introducing dependence on a proprietary model's calibration.

Relevance to Vision-Language Models

WorkDrive directly targets a key failure mode of VLMs in autonomous driving: the inability to override pre-training biases with domain-specific causal reasoning when visual priors break down. The paper's finding that VLMs can perceive work-zone objects yet still produce incorrect trajectories — because they lack the causal link between those objects and planning decisions — has broad implications for deploying VLMs in any OOD context. The perception-grounded annotation paradigm (external vision models → structured facts → LLM causal annotation) is a transferable recipe for constructing reasoning supervision in domains where self-generated rationales are unreliable. The two-stage SFT + GRPO training with a consistency-based RL reward extends the AlphaDrive/DriveR1 line of VLM alignment work to OOD evaluation, providing evidence that reasoning-trajectory consistency is a learnable and reward-able signal beyond curated in-distribution datasets.