Skip to content

iFinder: Structured Zero-Shot Vision-Based LLM Grounding for Dash-Cam Video Reasoning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

iFinder is a training-free, modular pipeline that decouples perception from reasoning for dash-cam video analysis by converting raw video into a hierarchical JSON data structure and feeding it to a general-purpose LLM via a three-block prompt. It outperforms both generalist and driving-specialized V-VLMs on four zero-shot benchmarks, gaining up to 39% on accident reasoning accuracy over the best driving-specialized baseline.

Problem

End-to-end video-VLMs (V-VLMs) lack structured inductive biases for driving-domain reasoning: they fail at spatial localization, causal inference (e.g., lane-cutting maneuvers, ego-vehicle approach rate), and fine-grained scene understanding. Additionally, dash-cam deployments often have only monocular RGB video—no LiDAR, GPS, or CAN bus—making robust post-hoc analysis from vision alone an open challenge. Existing domain-specific models (DriveMM, WiseAD) are still trained end-to-end and do not generalize to diverse conditions.

Method

iFinder maps dash-cam video V to a structured data object D via eight sequential pretrained vision modules: 1. Lens undistortion — GeoCalib + OpenCV to rectify camera intrinsics/distortion coefficients. 2. Scene understanding — InternVL (image-VLM) for static context; VideoLLaMA2 (video-VLM) for temporal event captions. 3. Ego-state estimation — DROID-SLAM camera-pose outputs converted to human-readable turning/motion labels via heading-angle deltas and speed thresholds. 4. 2D detection + tracking — OWL-V2 detector + ByteTracker, assigning persistent object IDs. 5. Lane assignment — OMR lane detector; objects assigned to lane sections by bounding-box bottom midpoint. 6. Metric depth/distance — Metric3D depth map + SAM segmentation mask; per-object distance = mean of masked depth region. 7. Object attributes — InternVL prompted on cropped object patches for color, traffic-light state, etc. 8. 3D orientation — CenterTrack 3D detector; yaw angles transferred to tracked 2D objects via Hungarian matching on projected boxes.

D is organized into a hierarchical JSON (video-level: global context, ego state, peer V-VLM response; frame-level: per-object {ID, class, bbox, distance, attributes, yaw, lane}). A three-block prompt (key explanation of D's schema → step-by-step reasoning instructions → peer-skepticism instruction) drives GPT-4o-mini to refine the peer V-VLM's initial response using verifiable symbolic cues.

Key Contributions

  • Training-free, modular grounding pipeline that converts monocular dash-cam video into an LLM-readable hierarchical data structure.
  • Eight-module perception stack extracting object pose (3D yaw), lane context, metric distance, ego-state, and scene semantics from pretrained models only.
  • Three-block prompting strategy combining key-explanation, step-instruction, and peer-instruction blocks for structured causal reasoning.
  • Zero-shot SOTA on four driving VQA benchmarks without any task-specific fine-tuning.
  • Ablation evidence that object orientation and scene understanding dominate accuracy over distance and lane location.

Results

  • MM-AU (multi-choice accident reasoning): iFinder 63.39% vs. best generalist VideoLLaMA2 52.89% (+10.5%) and best driving-specialized DriveMM 24.22% (+39.17%).
  • SUTD (6-category traffic QA): iFinder best across all six categories (Basic Understanding 52.2, Forecasting 43.5, Reverse Reasoning 50.2, Counterfactual 56.8, Introspection 39.2, Attribution 49.6); DriveMM is closest driving-specialized competitor.
  • LingoQA (open-ended VQA, Lingo-Judge accuracy): iFinder 44.20% vs. VideoChat2 41.20%, Video-LLaVA 21.00%, WiseAD 13.40%.
  • Nexar (accident occurrence prediction): iFinder 62.0% accuracy vs. VideoChat2 58.0%, VideoLLaMA2 50.0%, DriveMM 49.0%; iFinder avoids the all-accident-prediction bias seen in VideoLLaMA2/WiseAD (recall=1.0).
  • Weather/lighting robustness (MM-AU): iFinder 75%/65.52%/57.86%/63.69% (foggy/snowy/sunny/rainy) vs. DriveMM 33.33%/24.14%/20.13%/24.55%.
  • Error propagation resilience: at Ď„=0.8 only 0.68% of objects retained, accuracy drops to 58.27%—still above best baseline (52.89%).

Limitations

  • Not designed for real-time deployment; inference involves 8+ sequential pretrained models with no latency optimization.
  • Relies on camera intrinsics estimation (GeoCalib) being accurate; poor calibration propagates through the full pipeline.
  • 3D orientation estimation (CenterTrack) is assessed only indirectly; ablation shows 4.5% drop without it but the 3D detector's failure modes are not analyzed.
  • BLEU/METEOR on LingoQA are slightly below VideoChat2, suggesting lexical fluency is sacrificed for factual accuracy.
  • All experiments use a single LLM backend (GPT-4o-mini); sensitivity to LLM choice is not evaluated.
  • Post-hoc analysis scope only: not applicable to online/reactive driving systems.

Relevance to Vision-Language Models

iFinder directly challenges the dominant end-to-end V-VLM paradigm by showing that symbolic, structured intermediate representations outperform implicit visual token representations for spatial and causal reasoning in a specialized domain. For VLM researchers, this highlights a persistent weakness of current video-LLMs—their inability to reliably ground reasoning in metric spatial facts (distance, orientation, lane)—and proposes decoupled perception–reasoning as a complementary or corrective design pattern. The peer-instruction prompting block, where the LLM is explicitly told to distrust and correct a peer V-VLM's output, is a lightweight technique immediately applicable to other agentic VLM pipelines. The work also frames driving video understanding as a strong evaluation domain for VLM spatial reasoning, with four public benchmarks now validated as discriminative testbeds.