Skip to content

GHR-VLM: Making Zero-Shot Transit Video Analytics Realizable with Grounded Hybrid Reasoning

🕒 Published (v1): 2026-07-15 08:09 UTC · Source: Arxiv · link

Why this paper was selected

Zero-shot transit video analytics with grounded hybrid VLM reasoning; practical deployment recipe

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

GHR-VLM is a zero-shot transit bus video analytics framework that chains lightweight edge-side perception (door detection, passenger tracking via SAM 3) with selective cloud-side VLM inference on compacted, spatiotemporally grounded evidence clips. It avoids task-specific training by decomposing the hard problem of payment-method classification into geometry-aware sub-tasks that a general VLM can handle. Evaluation on 486 minutes of real bus footage demonstrates the viability—and current limits—of this grounded hybrid paradigm.

Problem

Supervised video models require task-specific annotated data that is costly and sparse for onboard bus payment analytics (QR, cash, tap, swipe, fare evasion). Direct VLM prompting on long surveillance streams is unreliable—payment events are brief, spatially small, visually degraded, and easily missed in long-context video—and fleet-scale cloud inference is prohibitively expensive.

Method

GHR-VLM operates in four cascaded stages under an edge-cloud split:

  1. Door-Grounded Stop Filtering: SAM 3 localizes the front-door region once and caches the bounding box \(B_d\). A VLM analyzes 12-frame overlapping 10-second windows of the cropped door stream to predict door state transitions \(z_\ell \in \{c{\to}o, o{\to}c\}\); a second VLM call verifies each candidate, yielding stop intervals \(S_k = [t_k^o, t_k^c]\).

  2. Rule-Based Passenger Tracking: Within each \(S_k\), SAM 3 tracks passengers in 16-frame batches with cross-batch IoU re-identification (\(\theta_I = 0.20\)). The primary passenger per frame is selected by bounding-box area as a projective camera-geometry prior (\(m_i = \arg\max_j \operatorname{area}(B_{i,j})\)). A two-step microfill/consistency repair smooths identity switches.

  3. Complex-to-Simple (CS) Direction Mapping: Rather than asking the VLM to classify "boarding vs. alighting" (domain-specific), it is prompted only for facing orientation \(\hat{o}_{k,j} \in \{\text{front, side, back, inside}\}\). A fixed rule \(\psi_\text{CS}\) maps orientation to direction, exploiting the fixed inward-facing camera geometry to avoid domain knowledge injection into the VLM prompt.

  4. Two-Stage Coarse-to-Fine Payment Classification: SAM 3 localizes the farebox region \(B_f\) once via the text prompt "farebox payment box." Stage 1 samples 16 frames uniformly from the full passenger clip, crops them with a broader margin \(B_f^{(1)}\), arranges them in a \(4\times4\) contact sheet \(G_{k,j}^{(1)}\), and asks the VLM \(\Phi_\text{pay}\) for a provisional label plus an evidence frame set \(\mathcal{E}_{k,j}\). Stage 2 resamples 16 frames from the temporal interval spanned by \(\mathcal{E}_{k,j}\), crops with a tighter \(B_f^{(2)}\), and issues a final classification call \((\hat{y}_{k,j}, \hat{c}_{k,j}, \hat{\rho}_{k,j})\).

Key Contributions

  • Grounded hybrid reasoning framework: explicit spatiotemporal grounding (detector + tracker + SAM 3) feeds compact evidence to a VLM, enabling zero-shot payment analytics without payment-annotated training data.
  • Complex-to-Simple (CS) mapping that reformulates domain-specific activity recognition into a generic visual-orientation judgment, decoupling VLM reasoning from transit-domain knowledge.
  • Two-stage coarse-to-fine farebox grounding: Stage 1 discovers evidence frames; Stage 2 temporally and spatially concentrates resampling onto those frames to surface fine-grained payment cues.
  • End-to-end bus payment data collection pipeline producing stop-level, passenger-level, and payment-level structured records from a single surveillance stream.
  • Real-world dataset of 486 minutes of annotated bus surveillance video (daytime and nighttime conditions, two recording sessions).

Results

  • Stop detection F1: 0.887 (C3_1) and 0.898 (C3_3) after Round 2 refinement, up from 0.767 and 0.708 in Round 1.
  • Passenger-clip detection F1: 0.702 (C3_1) and 0.847 (C3_3) after Round 2.
  • Five-class payment accuracy: Stage 2 improves C3_3 from 0.485 → 0.536; Stage 2 degrades C3_1 from 0.349 → 0.313.
  • Binary evasion detection accuracy: C3_1: 0.813 (Stage 1) → 0.783 (Stage 2); C3_3: 0.837 (both stages).
  • Evasion recall: C3_3 improves 0.559 → 0.647 with Stage 2; C3_1 drops 0.491 → 0.358.
  • Swipe recall on C3_3 rises from near-zero to 0.333 after Stage 2 refinement, showing benefit of focused temporal/spatial evidence for subtle payment gestures.
  • No supervised VLM fine-tuning baselines are compared; the paper positions against model-based pipelines qualitatively rather than numerically.

Limitations

  • Overall five-class payment accuracy (0.313–0.536) is insufficient for practical deployment, as stated by the authors.
  • Stage 2 refinement is conditionally harmful: it degrades performance on C3_1 where illumination is poor (direct sunlight, shadows), suggesting the coarse-to-fine strategy is sensitive to upstream visual quality.
  • The system inherits error propagation from edge modules—missed or wrongly filtered passenger clips cannot be recovered at the VLM stage.
  • No comparison against fine-tuned video VLM baselines or alternative zero-shot VLM pipelines; generalization of claims is limited.
  • Edge-module accuracy drops at night and under occlusion; atypical events (driver leaving cockpit, simultaneous boarding, passenger reappearance) cause systematic failures.
  • Single fixed-camera bus deployment; transferability to other transit vehicle types or camera configurations is untested.
  • The dataset is small (486 min, 2 video sequences, one bus line) and does not support statistical significance claims.

Relevance to Vision-Language Models

This paper directly probes a practical failure mode of VLMs—attention dilution and spatial imprecision in long, low-quality video—and proposes visual grounding as a structural remedy rather than a prompting trick, which is relevant to the broader question of when and how VLMs should be composed with classical perception. The CS mapping contribution is a general design pattern for bridging domain-specific semantics and VLM-native visual reasoning, applicable beyond transit. The two-stage evidence-selection prompting (asking the VLM to identify supporting frames, then re-querying at higher spatial resolution) is a reusable technique for improving VLM precision on spatially small or temporally sparse targets. Results quantify the gap between zero-shot VLM capability and deployment-grade accuracy in a real-world fine-grained recognition task, informing expectations for VLM-based perception systems.