Skip to content

CoC-VLA: Delving into Adversarial Domain Transfer for Explainable Autonomous Driving via Chain-of-Causality Visual-Language-Action Model

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CoC-VLA proposes an adversarial sim-to-real domain transfer framework for explainable end-to-end autonomous driving, pairing a teacher VLM (trained on simulated long-tail scenarios) with a student VLM (trained on real-world data) connected via a GAN-style discriminator. A Chain-of-Causality (CoC) architecture integrates temporal context through compressed prior-frame summaries rather than full sensor histories, enabling chain-of-thought reasoning from perception through prediction to ego-vehicle planning. On nuScenes-VLM, CoC-VLA outperforms DriveLM on both VQA metrics and open-loop planning.

Problem

Existing LLM-based autonomous driving approaches train on either real-world data—which lacks coverage of rare, long-tail events (accidents, non-compliant behaviors)—or simulation data that cannot be directly deployed in the real world. No prior method effectively transfers long-tail handling capability from simulation to real-world deployment within a single VLM-based end-to-end framework. Additionally, feeding multi-frame image sequences directly into an LLM inflates token length and compute cost.

Method

The framework has three components sharing a common backbone (LLaVA-v1.5 + LLaMA-7b):

  1. CoC VLM (shared base): Four modules—a Text Adapter that concatenates a compressed summary of the previous frame's CoC answer with the current end-to-end prompt (reducing token cost vs. raw sensor history), a Vision Adapter using CLIP + MLP projection over stacked 6-camera surround images, an LLM Brain (LLaMA) for chain-of-thought reasoning, and a CoC Answer module producing a causal chain covering perception→prediction→planning, with a filter that distills it into a one-line temporal instruction for the next step.

  2. Teacher VLM: Pre-trained on CARLA simulation data to handle hard/rare scenarios.

  3. Student VLM: Pre-trained on real-world nuScenes data; refined via adversarial training.

Adversarial training (2 steps): - Step 1: Both VLMs forward-pass their respective domain data; extracted features feed a transformer-based discriminator, which is updated via its own loss while VLM parameters are frozen. - Step 2: Student VLM is updated using its driving loss plus discriminator loss (gradients flow through discriminator but do not update it). This min-max formulation (Eq. 11) minimizes an f-divergence upper bound between source and target feature distributions, parameterized by a learned transformation matrix T_{s2r}.

Key Contributions

  • First VLM-based autonomous driving model that adversarially transfers long-tail scenario handling from simulation to real-world deployment.
  • Novel transformer-based discriminator that learns the sim-to-real feature distribution gap and guides student VLM alignment.
  • Backpropagation strategy (selective parameter freezing per step) for stable adversarial convergence.
  • Chain-of-Causality Text Adapter that uses compressed prior-frame LLM outputs instead of raw historical sensor tokens, cutting token length while preserving temporal causality.
  • Evaluation on nuScenes-VLM across VQA and open-loop planning metrics against DriveLM and ablation variants.

Results

VQA on nuScenes-VLM (vs. DriveLM baseline): - BLEU-4: 58.84 vs. 55.13 (+3.71) - ROUGE-L: 71.70 vs. 68.32 (+3.38) - CIDEr: 23.21 vs. 3.62 (~5× improvement) - Accuracy: 62.64 vs. 58.21 (+4.43, ≈1.05× relative) - Match: 45.30 vs. 36.67; SPICE: 51.88 vs. 34.59

Open-loop planning on nuScenes-VLM: - ADE: 1.44 vs. DriveLM 1.71 (−15.8%) - Collision rate: 1.58% vs. DriveLM 1.87% (−15.5%) - ADE improvement of 0.16 over sequential fine-tune (FinetuneTwice) ablation

Ablation: teacher VLM alone (sim-trained, real-tested) achieves ADE 17.33—confirming the necessity of the adversarial transfer step.

Limitations

  • Evaluated only on nuScenes-VLM (open-loop); no closed-loop evaluation in real-world or high-fidelity simulation.
  • CARLA simulator used as the source domain; sim-to-real gap may vary significantly for other simulators or geographic domains.
  • Inference deploys only the student VLM, but the full training pipeline requires separate pre-training of two large VLMs plus adversarial fine-tuning, making training resource-intensive.
  • Vision Adapter stacks six surround-view images without extrinsic/intrinsic calibration or BEV transformation; spatial reasoning may be limited compared to geometry-aware methods.
  • No comparison against more recent VLA baselines (e.g., DriveVLM, DriveMLM) on planning metrics in Table 2.

Relevance to Vision-Language Models

CoC-VLA directly extends the VLM paradigm (LLaVA-v1.5) into a domain-adaptive Vision-Language-Action setting, demonstrating that adversarial distribution alignment can bridge sim-to-real gaps in VLM-driven perception-planning pipelines—a problem not addressed by standard VLM fine-tuning. The Chain-of-Causality design offers a practical strategy for compressing temporal context into language tokens, relevant to any VLM application requiring sequential grounding without quadratic token scaling. For researchers tracking VLMs, this work shows that the teacher-student adversarial paradigm generalizes beyond vision classification to structured, multi-step natural language reasoning over action sequences. It also highlights the gap between VQA-style VLM evaluation and downstream planning performance, a recurring tension in autonomous driving VLM research.