Skip to content

Where Does It Exist from the Low-Altitude: Spatial Aerial Video Grounding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces Spatial Aerial Video Grounding (SAVG), a new task for localizing a referred object's spatio-temporal tube in UAV aerial videos via natural language queries. The authors contribute UAV-SVG, a large-scale benchmark with 2M+ frames and 216 object categories, and propose SAVG-DETR, an end-to-end transformer that decouples multi-modal multi-scale encoding and employs hierarchical progressive decoding with offset-based anchor refinement. SAVG-DETR outperforms all existing SVG baselines on the new benchmark.

Problem

Existing Spatial Video Grounding (SVG) benchmarks (VID-sentence, VidSTG, HC-STVG) are limited to ego-centric, ground-level, fixed-perspective videos with few object classes and large object sizes. UAV-based SVG is unexplored despite its practical importance (surveillance, delivery, patrol). Aerial scenes introduce extreme small-object sizes (1–200 pixels), dense clutter, illumination/occlusion variation, rapid viewpoint changes from UAV motion, and high frame counts—challenges that break existing SVG architectures, particularly when multi-scale features are naively combined with multi-modal transformers due to quadratic attention costs.

Method

SAVG-DETR is an end-to-end transformer with three innovations:

  1. Decoupled multi-modal multi-scale encoder: Instead of running self-attention over all modalities and scales jointly (prohibitively expensive), the encoder splits into (a) an intra-scale multi-modality interaction branch that performs cross-modal attention only on the high-level (S5) visual features paired with RoBERTa language features using learnable frame tokens (spatial) and video tokens (temporal), and (b) a cross-scale visual-only fusion branch that fuses S3/S4/S5 without language, using MSFF-FE, frequency-focused downsampling (FD), and semantic alignment & calibration (SAC) modules borrowed from real-time detection transformers.

  2. Multi-level progressive spatial decoder (MPSD): Decodes from high-level to low-level features in a 6-layer cascade (1 High + 2 Mid + 3 Low decoders). A multi-level language modulation module uses cross-attention to update language features with each visual scale, yielding level-specific language features that guide queries at each decoder stage.

  3. Offset-based anchor refinement (Query and Position Generator): Instead of predicting absolute boxes, each decoder layer predicts 4D offsets from context-rich object queries and iteratively updates reference anchor boxes. Initial queries come from the video token; initial anchors come from frame tokens modulated by the video token via learned scale (γ) and shift (β) factors. This coarse-to-fine paradigm improves cross-frame consistency. Larger auxiliary bounding boxes are used in the loss for better small-object supervision.

Key Contributions

  • Introduces the SAVG task: spatial tube grounding from natural language in UAV aerial video.
  • UAV-SVG benchmark: 3,564 videos, 2.01M frames, 17,820 video-sentence-tube triples, 216 object categories, 18.7h total duration; annotations generated via Gemini 1.5 + manual correction from WebUAV-3M tracking data.
  • Decoupled encoder that avoids quadratic blowup from multi-modal multi-scale joint attention while maintaining performance.
  • MPSD decoder that progressively refines from coarse high-level to fine low-level features, with level-specific language modulation.
  • Offset-based anchor generation from video/frame tokens for temporally consistent tube prediction.
  • New evaluation metric m_fAcc / fAcc@R (per-frame accuracy across a video) to assess localization stability beyond global vIoU.

Results

  • On UAV-SVG test set, SAVG-DETR achieves m_vIoU 27.15, vIoU@0.3 38.18, vIoU@0.5 22.85, m_fAcc 28.82, fAcc@0.3 35.85, fAcc@0.5 26.55.
  • Best prior method (STCAT, NeurIPS'22): m_vIoU 24.14, m_fAcc 27.17 — SAVG-DETR improves by +3.01 m_vIoU and +1.65 m_fAcc.
  • VideoGrounding-DINO (Swin-Trans backbone, CVPR'24): m_vIoU 23.83 — outperformed by +3.32 m_vIoU despite SAVG-DETR using only ResNet101.
  • Ablation: full model vs. IMIB-only baseline: +4.77 m_vIoU; adding CVFB: +6 points over naive multi-scale decoder; QPG adds ~1.3 points.
  • Efficiency: SAVG-DETR (203.08 GFLOPs) reduces FLOPs by 17% vs. Variant D (245.37 GFLOPs) while gaining +0.9% m_vIoU; CG-STVG uses 231.31 GFLOPs at 21.23 m_vIoU.

Limitations

  • Larger model size and higher memory usage (28.7 GB) than simpler baselines (e.g., SGFDN at 4.3 GB), limiting deployment on edge UAV hardware.
  • Visual backbone is ResNet101 initialized from MDETR pretrained on standard image datasets; no large-scale aerial pretraining is used.
  • Benchmark construction relies on Gemini 1.5 for referring expression generation, which may introduce language biases or hallucinations not fully caught by manual cleaning.
  • Evaluation is on a single new benchmark (UAV-SVG); generalization to other aerial domains (satellite, indoor drones) is not demonstrated.
  • Failure cases under extreme occlusion or when the target leaves the field of view are acknowledged but not quantified.

Relevance to Vision-Language Models

This paper extends vision-language grounding to a challenging new domain (UAV aerial video) where existing VLM-based grounding architectures fail due to extreme small-object scales, dynamic viewpoints, and prohibitive multi-scale attention costs. The decoupled encoder design offers a practical template for scaling cross-modal transformers to dense multi-scale settings without quadratic complexity—a broadly relevant architectural insight for VLMs used in spatial reasoning. The language modulation module (injecting multi-scale visual context into language features) and offset-based anchor refinement are techniques applicable to any VLM grounding pipeline that must maintain temporal consistency. The UAV-SVG benchmark also exposes a gap in current VLM evaluation: top models like VideoGrounding-DINO underperform significantly in aerial conditions, suggesting VLMs optimized for human-centric ground-level data do not transfer well to non-standard perspectives.