Skip to content

\(AutoDrive\text{-}P^3\): Unified Chain of Perception-Prediction-Planning Thought via Reinforcement Fine-Tuning

🕒 Published (v1): 2026-03-30 07:28 UTC · Source: Arxiv · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; chain of perception-prediction-planning with RL for autonomous driving VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AutoDrive-P³ is an end-to-end VLM-based autonomous driving framework that unifies perception, prediction, and planning through a structured Chain-of-Thought (CoT) reasoning pipeline. It introduces \(\text{P}^3\text{-GRPO}\), a hierarchical reinforcement fine-tuning algorithm that applies multi-component reward supervision jointly across all three stages rather than the planning-only supervision used by prior work. The system achieves state-of-the-art planning performance on nuScenes (open-loop) and NAVSIMv1/v2 (closed-loop) benchmarks.

Problem

Current VLM-based autonomous driving systems suffer from two compounding failures: (1) some models output trajectories directly without CoT reasoning, skipping perception and prediction entirely and creating a large domain gap; (2) others answer perception/prediction/planning queries but treat them as separate fragmented tasks with no information flow between stages. Additionally, existing GRPO-based driving approaches apply reinforcement optimization only to planning outputs, leaving perception and prediction without reward signal and producing superficial gains with limited interpretability.

Method

AutoDrive-P³ operates in two training phases on top of a Qwen2.5-VL-3B backbone:

Phase 1 — SFT cold-start. The \(\text{P}^3\text{-CoT}\) dataset (25,303 nuScenes frames + 115,434 NAVSIM frames) is constructed by: annotating critical objects with 2D bounding boxes (perception labels), projecting their future trajectories (prediction labels), and recording the ego trajectory (planning labels). Qwen2.5-VL-72B then generates coherent CoT chains that explicitly chain the three stages, with manual verification. SFT trains the model to produce structured outputs \(y = [y_\text{perception}; y_\text{prediction}; y_\text{planning}]\), each of the form \([\text{thinking}; \text{answer}]\), via standard NLL loss.

Phase 2 — \(\text{P}^3\text{-GRPO}\) reinforcement fine-tuning. GRPO samples \(G\) response groups and updates via a clipped surrogate loss with KL penalty. The reward is a weighted sum of four components: - \(R_\text{format}\): structural compliance. - \(R_\text{perc}\): average IoU × precision/recall of detected bounding boxes (Eq. 8). - \(R_\text{pred}\): IoU-weighted behavior-label accuracy × detection quality (Eq. 9). - \(R_\text{plan}\): \(\frac{2}{1 + e^{\text{clip}(L2)}}\) trajectory quality (Eq. 10).

Overall reward: \(R = \lambda_\text{format} R_\text{format} + \lambda_\text{perc} R_\text{perc} + \lambda_\text{pred} R_\text{pred} + \lambda_\text{plan} R_\text{plan}\).

Two inference modes are provided: detailed thinking (full CoT) and fast thinking (compressed reasoning).

Key Contributions

  • AutoDrive-P³ framework: end-to-end VLM architecture where perception outputs feed prediction, and both feed planning through a unified sequential CoT format.
  • \(\text{P}^3\text{-CoT}\) dataset: 140K+ annotated frames across nuScenes and NAVSIM with coherent three-stage CoT reasoning chains linking perception, prediction, and planning.
  • \(\text{P}^3\text{-GRPO}\) algorithm: hierarchical GRPO variant that simultaneously optimizes all three stages via modality-specific reward components, establishing causal reward dependencies matching the P→Pred→Plan chain.
  • Dual thinking modes: detailed vs. fast inference to trade off interpretability against latency.

Results

nuScenes (L2 displacement error ↓, collision rate ↓): - AutoDrive-P³ (Detailed): Avg L2 = 0.33 m, Avg collision = 0.06% - AutoDrive-P³ (Fast): Avg L2 = 0.34 m, Avg collision = 0.08% - Best prior VLM baseline OmniDrive/OpenDriveVLA: Avg L2 = 0.33 m, Avg collision = 0.10% - AutoDrive-P³ achieves 0.00% collision at 1s (vs. 0.01–0.07% for all baselines)

NAVSIMv1 (PDMS ↑): - AutoDrive-P³ (Detailed): PDMS = 90.6, (Fast): 90.2 - Best prior non-LiDAR baseline WoTE: PDMS = 88.3 (uses LiDAR); best image-only: DiffusionDrive = 88.1 - AutoDrive-P³ outperforms all listed baselines using only images (no LiDAR)

NAVSIMv2 (EPDMS, human-penalty-filter=True ↑): - AutoDrive-P³ (Detailed): EPDMS = 89.9, (Fast): 88.7 - WoTE (LiDAR-assisted): 88.2; DiffusionDrive: 87.7

Limitations

  • Uses only the front-view camera on nuScenes (6 frames over 3 seconds, 448×252 resolution), foregoing surround-view inputs available to many baselines.
  • Built on a 3B-parameter VLM; the CoT annotation pipeline depends on a 72B model (Qwen2.5-VL-72B), which may not be accessible at scale.
  • The \(\text{P}^3\text{-CoT}\) dataset is constructed from nuScenes and NAVSIM only; generalization to other domains (e.g., CARLA, Waymo) is not evaluated.
  • Detailed thinking mode has higher inference latency relative to fast thinking; the absolute latency figures are not reported.
  • Closed-loop evaluation still uses a simulation proxy (NAVSIM); real-world driving validation is absent.

Relevance to Vision-Language Models

This paper directly advances VLM adaptation for embodied control by demonstrating that structured, multi-stage CoT reasoning — enforced at training time through stage-specific reward signals — produces substantially better action quality than planning-only GRPO fine-tuning or fragmented QA supervision. The \(\text{P}^3\text{-GRPO}\) algorithm is a concrete recipe for extending GRPO (originally developed for LLM math reasoning) to settings with hierarchical, causally-dependent subtasks, a pattern broadly applicable to other VLM agent pipelines beyond driving. The dual thinking modes connect to the growing literature on inference-time compute scaling in VLMs. For VLM researchers, the key finding is that perception and prediction reward signals are not merely auxiliary — they are necessary for reliable downstream planning, validating the importance of grounded intermediate representations in VLM reasoning chains.