Skip to content

DriveAgent-R1: Advancing VLM-based Autonomous Driving with Active Perception and Hybrid Thinking

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

DriveAgent-R1: active perception and hybrid thinking for autonomous driving VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DriveAgent-R1 is a 3B-parameter VLM-based autonomous driving agent that replaces passive text-only reasoning with active perception: it proactively invokes a vision toolkit mid-reasoning to resolve visual uncertainty. A hybrid-thinking framework lets it switch adaptively between efficient text-only and tool-augmented reasoning, trained via a three-stage Cascaded RL strategy. With only 3B parameters, it matches GPT-5 and near-human performance on high-level behavioral planning benchmarks.

Problem

Existing VLM-based driving agents use passive perception: they reason purely over text descriptions of visual inputs and cannot seek additional visual evidence when the default view is ambiguous. Conversely, naively feeding all multi-view inputs is computationally wasteful and distracting. No prior work has instantiated tool-based, active multi-modal Chain-of-Thought (M-CoT) for high-level behavioral planning, nor enabled adaptive switching between text-only and tool-augmented reasoning modes in a driving context.

Method

Architecture. DriveAgent-R1 is built on Qwen2.5-VL-3B and augmented with a Vision Toolkit containing four tools: (1) Retrieve View — fetches arbitrary camera views or historical frames from a 5-second buffer; (2) RoI Inspection — crops and up-samples a bounding-box region from a high-resolution image; (3) Depth Estimation — produces a depth map for 3D spatial awareness; (4) 3D Object Detection — open-vocabulary 3D localization.

Hybrid Thinking. The model first emits a mode-selection token (<think_text> or <think_tool>). In tool mode the context history is updated iteratively: $\(H_k = H_{k-1} \oplus T_k \oplus I_k, \quad k < K\)$ where \(T_k\) is a generated tool call and \(I_k\) is the resulting encoded visual embedding.

Three-Stage Progressive Training: 1. Dual-Mode SFT (DM-SFT). Cold-starts with 4K high-quality CoT samples split evenly between tool-necessary (\(D_\text{tool}\), 2K) and tool-unnecessary (\(D_\text{text}\), 2K) cases, annotated by Qwen2.5-VL-72B and validated by a judge model. 2. Forced Contrastive Mode RL (FCM-RL). Uses Mode-Partitioned GRPO (MP-GRPO): for each query, forces generation of \(G/2\) text-mode and \(G/2\) tool-mode responses, aggregates all \(G\) into a single group for advantage normalization, enabling both intra-mode and inter-mode contrastive learning signals. 3. Adaptive Mode Selection RL (AMS-RL). Adds a conditional tool-usage reward \(R_\text{tool}\) that rewards a tool-based trajectory only if its accuracy exceeds the mean text-only accuracy in the group by a margin: \(R_\text{tool} \propto (R_\text{acc} - \bar{A}_\text{text} - \text{margin})\), penalizing superfluous tool calls.

Domain Alignment. Prior to planning training, Qwen2.5-VL-3B is fully fine-tuned on a 530K driving VQA dataset (scene description, entity recognition, localization, traffic rules) to ground visual sensitivity — yielding DriveAlign-3B as the initialization.

Key Contributions

  • First active-perception framework for VLM high-level driving planning: the agent proactively invokes a vision toolkit to ground decisions in verifiable visual evidence.
  • Hybrid-thinking framework integrating text-only and tool-augmented M-CoT with adaptive mode selection, enabled by a novel Cascaded RL training strategy.
  • Mode-Partitioned GRPO (MP-GRPO): forces balanced exploration of both modes per query, enabling inter-mode contrastive learning without mode collapse.
  • Conditional tool-usage reward that explicitly penalizes superfluous tool calls, incentivizing precision in active perception.
  • 3B-parameter model competitive with GPT-5 and human-level proficiency on long-tail driving scenarios, with reduced inference latency vs. passive multi-view approaches.

Results

  • Drive-Internaltest (First-Frame Joint Acc.): DriveAgent-R1 w/ tools achieves 51.34%, vs. GPT-5 56.48%, human 49.59%, GPT-4.1 43.18%; +6.07pp gain from tool access (largest among all tested models).
  • nuScenestest (Seq. Avg. Joint Acc.): DriveAgent-R1 achieves 47.10%, surpassing GPT-5 (45.14%) and approaching human (48.24%).
  • DriveBench: Perception 34.07 (vs. DriveLM 16.85, Dolphins 9.59); Behavior 43.69 (vs. DriveLM 42.78).
  • nuScenes open-loop motion planning (ADE avg ↓): DriveAgent-R1 0.28 m, vs. OpenDriveVLA-3B 0.33 m, OmniDrive 0.33 m, EMMA 0.32 m; best collision rate at 3s is 0.30%.
  • Smaller Qwen2.5-VL-3B and -7B baselines show performance degradation with tool access (e.g., -0.42 and -3.57 pp), demonstrating tool use is non-trivial without specialized training.

Limitations

  • Evaluated primarily on proprietary Drive-Internal data (35K clips); generalizability to fully public benchmarks is partially assessed via nuScenes only.
  • Maximum tool calls capped at 3 per inference step; complex multi-step scenes may require more interactions.
  • Motion planning head is a lightweight add-on MLP frozen on top of DriveAgent-R1; the architecture is not end-to-end trained for trajectory regression.
  • Comparison against GPT-5 relies on one-shot API prompting in "no-thinking" mode, which may not reflect GPT-5's full planning capability.
  • Vision Toolkit tools (depth estimation, 3D detection) are treated as oracle modules; their real-world error propagation under imperfect sensors is not analyzed.

Relevance to Vision-Language Models

DriveAgent-R1 directly advances VLM capability by replacing static single-pass visual encoding with an iterative, agentic loop — a significant step from passive to active multimodal reasoning. The Mode-Partitioned GRPO algorithm offers a concrete recipe for training VLMs to balance fast text-only inference against expensive tool-augmented perception, a pattern applicable beyond driving. The paper provides evidence that targeted domain alignment (530K VQA fine-tuning) and RL-based mode-selection training are necessary to prevent VLMs from ignoring high-dimensional visual inputs in favor of textual shortcuts — a finding relevant to any VLM deployment requiring reliable grounding. The hybrid-thinking paradigm connects to broader work on System-1/System-2 reasoning in LLMs, here instantiated as a learned, reward-shaped policy rather than a heuristic switch.