Skip to content

AUTOPILOT VQA: Benchmarking Vision-Language Models for Incident-Centric Dashcam Understanding

🕒 Published (v1): 2026-07-09 17:46 UTC · Source: Arxiv · Venue: CVPR · link

Why this paper was selected

CVPR dashcam VQA benchmark; incident-centric eval fills gap in autonomous driving VLM testing

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AUTOPILOT-VQA is an incident-centric VQA benchmark built on 600+ dashcam video clips annotated with 6,000+ structured question-answer pairs spanning environmental conditions, road context, and safety-critical incident attributes. It was released as part of the AUTOPILOT CVPR 2026 competition, attracting 59 teams and 686 submissions. Top models plateaued at ~0.66 mean accuracy, exposing a persistent gap between perceptual recognition and causal/temporal reasoning in current VLMs.

Problem

Existing autonomous driving benchmarks (KITTI, BDD100K, nuScenes, etc.) focus on routine conditions and low-level perception tasks; none systematically evaluate whether VLMs can reason about safety-critical incidents—fault attribution, impact characterization, avoidability, and agent interaction—from dashcam video. Prior VQA driving benchmarks do not cover the full incident-severity spectrum or provide open competitive evaluation.

Method

The benchmark wraps the VQA-Autopilot dataset of 600+ dashcam clips (27% collisions, 11% near-misses, 17% hazard-avoided, 27% no-incident) with a structured annotation schema of nine question groups (A–I), yielding 28 sub-questions per clip across: - Environmental conditions: weather, time of day, lighting - Road context: traffic environment, road/lane configuration, surface state, traffic control - Incident characterization: incident category, primary/secondary entity identity and behavior, fault attribution, prevention measures, impact location

Each clip requires integer-valued predictions mapped to predefined answer categories. Evaluation uses mean per-question accuracy \(\text{Score} = \frac{1}{Q}\sum_{q=1}^{Q} \text{Acc}_q\), with all 28 questions weighted equally. The benchmark is hosted as a Kaggle competition for open, reproducible comparison.

Key Contributions

  • Incident-centric VQA benchmark with structured annotations covering the full accident-severity spectrum and nine semantic question groups
  • Over 6,000 annotated question-answer pairs from 600+ real dashcam clips
  • Open competitive evaluation platform (Kaggle) enabling diverse community submissions under a shared scoring protocol
  • Empirical characterization of current VLM failure modes on safety-critical, causal, and temporally grounded reasoning

Results

  • Top leaderboard score: 0.65835 mean per-question accuracy
  • Second and third place: 0.65505 and 0.65371 respectively
  • Many submissions clustered near 0.39–0.40, indicating naive or partially optimized pipelines struggle broadly
  • Only a limited number of teams exceeded 0.60 accuracy
  • 224 registered entrants, 73 active participants, 59 teams, 686 total submissions
  • No external VLM baselines are reported; all numbers are from competition submissions

Limitations

  • No zero-shot or few-shot evaluation of named frontier VLMs (GPT-4V, Gemini, LLaVA, etc.) is reported; competition results reflect optimized pipelines of unknown architecture
  • Annotation is human-labeled but inter-annotator agreement statistics are not reported
  • Video clips are restricted to dashcam perspective; no multi-sensor or bird's-eye-view data
  • Unknown and non-applicable answer classes are present but their frequency and per-question impact are not quantified
  • Error analysis is qualitative; no per-category accuracy breakdown is provided
  • Dataset distribution is skewed toward daytime (70%) and dry roads (83%), potentially limiting generalization to adverse conditions

Relevance to Vision-Language Models

AUTOPILOT-VQA directly probes a known weakness of VLMs—causal, temporal, and relational reasoning beyond object recognition—in a high-stakes domain where errors have real consequences. The ceiling of ~0.66 accuracy despite competitive engineering effort quantifies the gap between current multimodal models and reliable safety-critical scene understanding. For researchers tracking VLMs, this benchmark complements perception-focused evaluations (DrivingVQA, MetaVQA, NuPlanQA) by isolating incident reasoning as a distinct capability axis. It also establishes a public leaderboard enabling longitudinal tracking of VLM progress on structured, grounded QA over video.