Skip to content

Teaching Vision-Language-Action Models What to See and Where to Look

🕒 Published (v1): 2026-07-02 03:34 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

VLA training data design for autonomous driving; addresses vision-text imbalance in VLA supervision

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DriveTeach-VLA addresses the spatial grounding deficit in Vision-Language-Action (VLA) models for autonomous driving, where text-centric training produces semantically rich but spatially ungrounded attention. The framework introduces Driving-aware Vision Distillation (DVD) to teach the vision encoder what traffic-critical objects to attend to, and 2D Trajectory-Guided Prompts (2D-TGP) to teach the planner where in the image feasible trajectories lie. Together, they achieve state-of-the-art results on NAVSIM Navtest (90.4 PDMS) and nuScenes (L2=0.30m, collision=0.12%).

Problem

Existing AD-VLA models rely on text-centric VQA pretraining and chain-of-thought (CoT) reasoning data that supervises linguistic semantics but not spatial or motion dependencies. As a result, ViT encoders (pretrained on natural images) fail to focus on driving-critical objects, and attention during trajectory decoding is scattered rather than grounded to actionable regions. This gap between perceived semantics and spatial-action grounding limits reliable trajectory prediction.

Method

DriveTeach-VLA is a dual-model pipeline built on Qwen2.5-VL-3B:

TGP-Prompter — DVD Pretraining (what to see): - An open-vocabulary detector (Grounding DINO) detects traffic-critical objects (cars, pedestrians, traffic lights, etc.) and overlays their bounding boxes on raw images to produce bbox-augmented images \(C_\text{bbox}\). - Self-distillation is applied to the ViT encoder: a teacher ViT processes \(C_\text{bbox}\), a student ViT processes the raw image \(C\). Block-wise alignment loss minimizes Smooth-L1 between averaged block features: $\(\mathcal{L}_\text{distill} = \frac{1}{K}\sum_{k=1}^{K}\text{Smooth-L1}\!\left(\bar{v}_k^t, \bar{v}_k^s\right)\)$ Teacher weights are updated via EMA. This injects traffic priors into the encoder without VQA pairs. - The TGP-Prompter decoder is additionally SFT-trained to predict 2D-TGP trajectories: expert BEV waypoints \((x_t, y_t)\) projected onto the image plane via a pinhole camera model \(\pi(K, [R|t], \cdot)\), encoded as text-form 2D coordinate sequences \(P_I\).

TGP-Planner — TGP-Guided SFT + GRPO (where to look): - Initialized from TGP-Prompter weights to inherit driving priors. - Conditioned on 2D-TGP keypoints \(\{(x_t^I, y_t^I)\}\) alongside image, ego-state, and CoT reasoning (pseudo-labeled by Qwen2.5-VL-72B). - Trained with SFT on CoT + trajectory prediction, then GRPO-RL using PDM-Score (PDMS) as reward, which factors No-at-fault Collisions (NC), Drivable Area Compliance (DAC), Ego Progress (EP), Time-to-Collision (TTC), and Comfort (C). - During training, teacher forcing supplies ground-truth 2D-TGP; at inference, TGP-Prompter generates the 2D-TGP first, then feeds it to TGP-Planner.

Key Contributions

  • Identifies that text-centric VLA pretraining produces spatially ungrounded attention; introduces Attention Mass (AM) as a quantitative metric for spatial grounding quality.
  • DVD: Bbox-augmented self-distillation that injects driving-specific visual priors into the ViT encoder without relying on VQA text supervision.
  • 2D-TGP: Projects BEV expert trajectories onto the image plane via the pinhole camera model, providing spatially aligned, MLLM-interpretable planning cues as textual 2D coordinates.
  • A three-stage pipeline (DVD pretraining → CoT-SFT with 2D-TGP → GRPO-RL with 2D-TGP) replacing the standard VQA pretraining stage.
  • SoTA on NAVSIM Navtest and nuScenes among VLA-based methods using only a 3B-parameter backbone.

Results

  • NAVSIM Navtest (PDMS):
  • DriveTeach-VLA: 90.4 PDMS (NC=98.5, DAC=96.9, TTC=97.9, EP=98.2)
  • AutoVLA (same backbone, Qwen2.5-VL-3B): 89.1 PDMS
  • CuriousVLA (Qwen2.5-VL-3B): 88.9 PDMS
  • ReCogDrive (InternVL3-8B, larger model): 89.6 PDMS
  • Best end-to-end non-VLA (ASSCG): 91.4 PDMS
  • nuScenes (UniAD metrics):
  • DriveTeach-VLA: L2=0.60m, Collision=0.31%
  • Impromptu VLA: L2=0.67m, Collision=0.38%
  • AutoVLA: L2=0.86m, Collision=0.35%
  • nuScenes (ST-P3 metrics):
  • DriveTeach-VLA: L2=0.30m, Collision=0.12% (ties UniAD on collision, best L2 among VLA models)
  • Ablation: removing DVD drops PDMS from 90.4 to ~88.x; \(\lambda_\text{TGP}=0.10\) gives best results; block-wise distillation (\(2\times4\)) outperforms patch-by-patch alignment.
  • Train–test gap from teacher forcing vs. TGP-Prompter inference: ~0.4 PDMS, deemed negligible.

Limitations

  • Dual-model inference overhead: Running TGP-Prompter before TGP-Planner adds latency compared to single-model VLAs; no inference-time cost analysis is reported.
  • Train–test distribution gap: Teacher forcing with ground-truth 2D-TGP during training vs. predicted 2D-TGP at inference introduces a mismatch, acknowledged but only partially characterized (~0.4 PDMS).
  • Single front-view camera: 2D-TGP projection is demonstrated for front-view only; multi-view generalization is not addressed.
  • Ground-plane assumption: The inverse projection \(\pi^{-1}\) requires \(z=0\); this fails for elevated or ramp driving scenarios.
  • Grounding DINO dependency: DVD pretraining quality is bounded by the off-the-shelf detector's accuracy on driving domains.
  • Non-reactive evaluation: NAVSIM Navtest is a non-reactive closed-loop benchmark; behavior under adversarial agents is untested.

Relevance to Vision-Language Models

This paper directly targets a well-known weakness in deploying VLMs for embodied tasks: their visual representations are shaped by language supervision and generalist pretraining, not by the spatial and action-level priors needed for grounded decision-making. The DVD self-distillation approach offers a domain-adaptation strategy for ViT encoders within VLMs that is label-efficient (no VQA annotation needed) and modular. The 2D-TGP mechanism is a concrete instance of a broader design principle—projecting structured, domain-specific signals into the token space that MLLMs already understand—which could generalize to other VLM-based agent tasks (robotic manipulation, navigation). For VLM researchers, the Attention Mass metric and the qualitative attention analysis provide a diagnostic tool for identifying spatial grounding failures in any MLLM fine-tuning regime.