Skip to content

ScanFocus: A Coarse-to-Fine Framework for Spatio-Temporal Video Grounding

🕒 Published (v1): 2026-07-15 03:50 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ScanFocus is a coarse-to-fine framework for Spatio-Temporal Video Grounding (STVG) that decouples the task into a sparse global scan and a dense local boundary refinement stage. It addresses the fundamental bottleneck of temporal boundary ambiguity caused by low-rate subsampling in prior methods by recovering suppressed high-frequency cues via the Semantic-Guided Temporal Aggregator (SGTA). Experiments on three benchmarks show state-of-the-art performance.

Problem

Existing Transformer-based STVG methods perform low-rate temporal subsampling to handle long videos affordably, which physically discards high-frequency boundary cues and often skips ground-truth boundary frames entirely. Additionally, these methods use an overloaded MDETR-style encoder that forces simultaneous tri-modal (appearance + motion + language) fusion, creating an optimization bottleneck. The result is temporally ambiguous predictions; an oracle experiment on HC-STVGv1 shows that replacing predicted timestamps with ground-truth ones raises vIoU@0.3 from 67.5% to 94.0%, confirming temporal localization—not spatial grounding—as the primary failure mode.

Method

ScanFocus operates in two cascaded stages sharing backbone weights:

Stage 1 — Global Spatio-Temporal Scan. \(T_c\) frames are uniformly sampled at a low rate \(f_c\). A unified Vision-Language Fusion encoder (BEiT-3) extracts pre-aligned appearance and linguistic features \(F_a, F_t\); a frozen VideoMAE provides motion features \(F_m\). A lightweight Deformable Semantic-Motion Fusion module fuses these via multi-scale deformable attention (Eq. 2) with linear complexity, then splits the output back into modality streams. Dual DETR-style decoders with semantic-injection query initialization produce coarse bounding-box tubes \(B_c\) and a coarse temporal interval \(\tau_c = (t_c^s, t_c^e)\) supervised by L1 + GIoU and KL-divergence losses (Gaussian-smoothed boundary targets).

Stage 2 — Local Boundary Focus. Dense windows of \(N_w\) frames are sampled at rate \(f_r = k_r \cdot f_c\) around \(t_c^s\) and \(t_c^e\). The shared encoder re-processes these dense clips. The Semantic-Guided Temporal Aggregator (SGTA) then: 1. Interpolates coarse boxes \(B_c\) to form a dense spatial prior \(B_{st} \in \mathbb{R}^{N_w \times 4}\). 2. Applies RoI Pooling to isolate object-aligned appearance features \(F_{ps}^a = \text{ROIPool}(F_{rs}^a, B_{st})\). 3. Computes semantic-guided motion: \(F_{sg} = (\phi_a(F_{ps}^a) \odot F_{rs}^m) \oplus \phi_t(F_{rs}^t)\). 4. Flattens \(F_{sg}\) into a sequence of length \(N_w \times N_m\) and passes through \(L_t\) temporal self-attention layers (Eq. 6), enabling full cross-frame interaction in the local window.

Refine decoders then predict per-frame boundary probabilities and action confidences via BCE loss. Coarse parameters are frozen during refine-stage training.

Key Contributions

  • ScanFocus framework: decouples STVG into global sparse retrieval and local dense boundary refinement, breaking the compute–accuracy trade-off.
  • Deformable Semantic-Motion Fusion: lightweight multi-scale deformable attention for cross-modal fusion that replaces the heavy MDETR-pretrained encoder, reducing the tri-modal optimization bottleneck.
  • Semantic-Guided Temporal Aggregator (SGTA): explicitly models short-term inter-frame dependencies within densely sampled boundary windows using RoI-guided appearance modulation and temporal self-attention, recovering high-frequency cues discarded by global subsampling.
  • Oracle analysis quantifying temporal localization as the dominant STVG bottleneck (vIoU@0.3: 67.5% → 94.0% with GT timestamps on HC-STVGv1).
  • Decoupled two-stage training preventing coarse-stage interference with boundary refinement gradients.

Results

(Numbers drawn directly from the paper text; full table data was truncated.)

  • HC-STVGv1 oracle gap: full model vIoU@0.3 = 67.5%, vIoU@0.5 = 42.2%, m_vIoU = 73.7%; GT-timestamp oracle vIoU@0.3 = 94.0%, vIoU@0.5 = 86.9% — motivating the refinement design.
  • Claims state-of-the-art on HC-STVGv1, HC-STVGv2, and VidSTG over prior Transformer-based STVG methods; full numerical tables were cut off in the provided text.

Limitations

  • Full quantitative comparison tables are absent from the provided text excerpt, so the magnitude of improvement over baselines cannot be confirmed from this text alone.
  • Dense boundary sampling doubles the inference-time forward passes (one global, two local windows), which may increase latency for real-time applications.
  • SGTA relies on coarse-stage bounding boxes for the spatial prior; large errors in \(B_c\) could propagate into inaccurate RoI features during refinement.
  • VideoMAE is used frozen for motion, which may limit adaptation to unusual motion domains not covered during VideoMAE pre-training.
  • The method is evaluated only on the standard STVG benchmarks (HC-STVG, VidSTG); generalization to longer or more complex videos (e.g., egocentric or multi-object) is not assessed.

Relevance to Vision-Language Models

ScanFocus directly advances VLM-grounded video understanding by replacing the MDETR-style pipeline—long the dominant fusion backbone in STVG—with a general vision-language encoder (BEiT-3) pre-trained on large-scale image-text data, demonstrating that task-specific pre-training can be supplanted by foundation-model alignment. The Deformable Semantic-Motion Fusion module shows how deformable attention can efficiently bridge static VLM representations with dynamic motion features without full encoder retraining. For researchers tracking VLMs, this paper is a concrete example of how to adapt image-text foundation models to temporal video tasks by pairing a frozen VLM backbone with a lightweight, task-specific temporal head—a design pattern increasingly relevant as VLMs are extended to video agents and embodied grounding. It also contributes an oracle methodology for diagnosing where VLM-based pipelines fail in multimodal grounding tasks.