Skip to content

What's Hidden Matters: Identifying Planning-Critical Occluded Agents using Vision-Language Models

🕒 Published (v1): 2026-07-01 00:14 UTC · Source: Arxiv · link

Why this paper was selected

VLMs for occlusion-aware autonomous planning; novel selective attention over hidden agents

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Autonomous vehicles face unequal occlusion risks, yet existing methods treat all hidden regions uniformly. This paper introduces a PKL-guided framework that uses Planning KL-Divergence to rank occluded agents by their impact on the ego-vehicle's trajectory, then leverages GPT-5 to generate structured semantic annotations, and fine-tunes VLMs on this curated data. Fine-tuned smaller models dramatically outperform zero-shot large models on the resulting nuScenes-based benchmark.

Problem

Current AV occlusion handling either applies uniform conservatism (causing "freezing" or overly defensive behavior) or predicts occluded regions generatively (unreliable, planning-unaware). No prior work systematically identifies which hidden agents are most critical to the ego-vehicle's immediate trajectory plan, nor trains VLMs specifically for this planning-aware occlusion reasoning task.

Method

The pipeline has three stages applied to nuScenes + Occ3D:

  1. Hidden agent extraction: Compute per-agent visibility score \(v_i = A_i^{\text{visible}} / A_i^{\text{bbox}}\) from voxel splatting; classify agents with \(v_i < \tau = 0.4\) as hidden.

  2. PKL-based ranking: For each hidden agent \(h_i\), compute its planning gain: $\(\text{Gain}(h_i) = \text{PKL}(V) - \text{PKL}(V \cup \{h_i\})\)$ where \(\text{PKL}(H) = \sum_{0 < \Delta \leq T} D_{\text{KL}}(p_\theta(x_{t+\Delta} \mid A_{\leq t}) \,\|\, p_\theta(x_{t+\Delta} \mid V_{\leq t}))\) measures trajectory distribution divergence caused by missing agents. Top-1,000 training / top-250 validation scenes are selected by highest \(\text{PKL}(V)\).

  3. Structured annotation via GPT-5: GPT-5 receives \(T=2\)s temporal multi-camera panoramas (6 cameras, 3 frames at 1 Hz) with the target agent's class and bounding box, and outputs structured JSON covering: visual cue type, camera view, agent class, action hypothesis, occluder class, occlusion type, priority level (High/Medium/Low), and BEV sector (e.g., Ahead-Left).

VLMs are fine-tuned with LoRA (rank 16, all linear layers) using ms-swift/AdamW (\(\text{lr}=2\times10^{-5}\), 4Ă—H100, 3 epochs), constrained to output structured JSON.

Key Contributions

  • First systematic framework for ranking occluded agents by planning impact using PKL gain as a data-curation signal, repurposing a perception-evaluation metric for dataset selection.
  • A new nuScenes-derived benchmark of 1,250 scenes (3,750 GPT-5-generated structured annotations) focused exclusively on planning-critical occlusions.
  • Comprehensive fine-tuning experiments across 20+ VLMs (Qwen2.5-VL, InternVL3.5, Gemma-3, MiniCPM, and domain-adapted DriveMM/DriveLM/DriveGPT4), with PKL-guided selection shown to improve over random sampling by ~30%.

Results

  • Fine-tuning yields average accuracy improvements of +0.09 to +0.45 across all models.
  • InternVL3.5-1B improves from 0.092 → 0.545 average accuracy (+493%); visual cue detection gains +950%, priority prediction +518%.
  • Qwen-3B-SFT (0.478 avg. accuracy) surpasses Qwen-72B zero-shot (0.440) while being 24Ă— smaller.
  • Domain-adapted baselines underperform: DriveMM achieves 0.001 avg. accuracy zero-shot; DriveLM reaches 0.098 avg. action score (best among domain-adapted, zero-shot).
  • PKL-guided data selection outperforms random sampling by ~30% on validation metrics.
  • PKL estimates stabilize after \(T=2\)s (~5 samples/scene), validating the chosen window length.

Limitations

  • Dataset is small (1,250 scenes, 3,750 annotations) and derived from a single dataset (nuScenes), limiting diversity and generalizability.
  • Ground-truth annotations are generated by GPT-5, introducing potential annotation noise and dependence on a proprietary model.
  • Evaluation is against GPT-5-generated labels rather than human-verified ground truth, creating a circular quality dependency.
  • The PKL planner is trained on nuScenes; its transfer to other AV stacks or sensor configurations is unvalidated.
  • Camera view localization shows the weakest gains post-fine-tuning (e.g., Qwen-3B-SFT: 0.196 accuracy), indicating persistent difficulty in spatial grounding.

Relevance to Vision-Language Models

This paper directly advances VLM adaptation for safety-critical perception tasks by demonstrating that planning-aware data curation—rather than scale or domain-specific pretraining—is the key driver of performance on structured occlusion reasoning. The PKL-guided annotation pipeline is a reusable recipe for injecting downstream task signals into VLM training data, applicable beyond driving. The finding that fine-tuned small VLMs outperform zero-shot large VLMs challenges assumptions about scale-driven generalization in specialized robotic perception. It also establishes a multi-field structured-output evaluation protocol that is more diagnostic than single-metric VLM driving benchmarks.