Skip to content

Aha! - Predicting What Matters Next: Online Highlight Detection Without Looking Ahead

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AHA is an autoregressive online highlight detection (OHD) framework that scores video frames in real time against a natural-language task description, processing frames causally without ever accessing future context. It combines a frozen SigLIP visual encoder, a Qwen2-based decoder, and three lightweight prediction heads (relevance, informativeness, uncertainty) with a novel Dynamic SinkCache for constant-memory streaming. AHA achieves state-of-the-art on TVSum (+5.9% mAP over prior best) and Mr.HiSum (+8.3% mAP@50), surpassing offline full-context methods.

Problem

Existing highlight detection methods—including transformer-based and VLM-based approaches—require access to the full video at inference time (offline, bidirectional attention), making them incompatible with streaming or real-time scenarios. The few streaming Video-LLMs that attempt highlight detection rely on post-hoc smoothing (implicitly using future frames), modified evaluation protocols, and achieve suboptimal detection performance. No prior method offers robust, task-conditioned, frame-level OHD on standard benchmarks under strict causal constraints.

Method

AHA is built on a frozen SigLIP visual encoder → single linear multimodal projector → Qwen2 decoder-only transformer pipeline. At each timestep, the decoder receives: (1) the current frame's visual tokens, (2) the fixed natural-language task objective Q and system prompt S (prepended once), and (3) the KV cache representing past context. Three lightweight linear heads on the decoder's final hidden state \(h_t\) predict: - Relevance (\(\hat{r}_t = W_r h_t\)): trained with Smooth L1 loss + total-variation regularizer against YouTube engagement scores. - Informativeness (\(\hat{i}_t = \text{softmax}(W_i h_t)\)): binary BCE loss, distinguishing novel vs. redundant frames. - Uncertainty (\(\hat{u}_t\)): log-variance of a Gaussian, trained with NLL + variance diversity penalty to prevent mode collapse. - An auxiliary LM head adds captioning supervision to enrich representations.

Final score \(\hat{y}_t\) combines all three heads via an uncertainty-aware piecewise function: relevance and informativeness are linearly weighted; high-uncertainty frames are penalized by \(\epsilon(\hat{u}_t - \tau_u)\).

Dynamic SinkCache: Modifies StreamingLLM's SinkCache so the persistent "sink" contains exclusively the task objective tokens Q (~45 tokens), while a 2048-token sliding window holds recent visual context. This fixes memory at 17% of an unbounded KV cache while outperforming it.

HIHD dataset: 22,463 YouTube videos with frame-level engagement scores (normalized replay counts), synthetically generated task queries from video titles, and quality-dropout augmentation (5–20% of frames corrupted with noise/blackout/resolution drops).

Key Contributions

  • AHA framework: causal, autoregressive OHD with three decoupled prediction heads and uncertainty-aware scoring, operating under strict no-future-access constraints.
  • Dynamic SinkCache: task-objective-focused sink + sliding visual window; constant memory footprint for arbitrarily long streams.
  • HIHD dataset: 22,463 videos with frame-level engagement-based labels, task conditioning, and quality dropout augmentation, released for OHD training and evaluation.
  • SOTA OHD: surpasses all prior offline and online methods on TVSum (93.0 mAP tuned, 91.6 zero-shot) and Mr.HiSum (64.19 mAP@50).
  • Robustness validation: quality dropout training yields graceful degradation under color banding (Δ0.4), block noise (Δ1.8), quality degradation (Δ4.1), and blackout (Δ4.8) on TVSum.
  • Real-world robotics application: 16/18 predicted peaks aligned with ground-truth navigation commands on the long-horizon SCOUT egocentric robot video.

Results

  • TVSum (zero-shot): 91.6 top-5 mAP, Kendall's τ = 0.304, Spearman's ρ = 0.433 — surpasses TR-DETR (87.1 mAP, tuned) and LLMVS.
  • TVSum (domain-adapted grid search): 93.0 top-5 mAP.
  • Mr.HiSum test set: 64.19 mAP@50, 32.66 mAP@15 — vs. PGL-SUM (55.89/27.45), DSNet (50.78/24.35), iPTNet (50.53/22.74).
  • Ablation — removing informativeness head: −9.8 mAP; removing relevance head: −15.7 mAP; removing uncertainty: −3.2 mAP; removing language conditioning: −11.8 mAP, Spearman's ρ drops from 0.406 → 0.342.
  • Memory ablation: Dynamic SinkCache (93.0 mAP) > Standard SinkCache (92.6) > Unbounded KV Cache (91.7) > Sliding Window Only (69.5) > Static Window Only (63.2).
  • Streaming moment retrieval (Charades-STA): 50.7% R@1 at IoU=0.5, 27.9% R@1 at IoU=0.7 — SOTA among streaming methods.
  • SCOUT robotics video: 16/18 detected peaks matched annotated events in an 8-minute clip.

Limitations

  • Uncertainty head lacks ground-truth uncertainty labels; trained purely on engagement-score deviation, limiting calibration in high-stakes settings.
  • HIHD engagement scores (YouTube replay counts) are an imperfect proxy for true importance and may encode engagement-bait biases.
  • Architectural ablations across different vision-language backbones were not performed due to compute constraints; backbone generalization is unvalidated.
  • Static inference weighting (α, β, ϵ, τ_u) is not adaptive; more sophisticated dynamic weighting was empirically inferior but unexplored at scale.
  • Training uses fixed-length windows with no persistent cross-segment memory; the Dynamic SinkCache is inference-only, potentially limiting long-range global reasoning.
  • SCOUT evaluation is qualitative and preliminary; no large-scale quantitative robotics benchmark exists.

Relevance to Vision-Language Models

AHA directly leverages VLM components (SigLIP encoder + Qwen2 decoder) in a streaming regime, demonstrating that a frozen visual backbone with a lightweight projection layer is sufficient for competitive frame-level understanding when trained with the right objectives. The Dynamic SinkCache offers a transferable architectural pattern for any VLM deployed on continuous video streams, where task-conditioned language tokens serve as persistent global context anchors. The paper shows that task language grounding is critical to online video understanding (+11.8 mAP without it), reinforcing the importance of natural-language conditioning in VLMs beyond static image tasks. This work extends the VLM streaming video literature (alongside VideoLLM-Online and StreamingLLM) by demonstrating that specialized lightweight heads atop a general-purpose VLM backbone can surpass purpose-built offline methods on their own benchmarks.