Skip to content

Unleashing the Potential of Multimodal LLMs for Zero-Shot Spatio-Temporal Video Grounding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper proposes a zero-shot framework for Spatio-Temporal Video Grounding (STVG) by exploiting a previously unreported property of MLLMs: certain special tokens (called "grounding tokens") dynamically attend to query-relevant visual regions. Two novel strategies—Decomposed Spatio-Temporal Highlighting (DSTH) and Temporal-Augmented Assembling (TAS)—are introduced to overcome the failure of these tokens to integrate complex attribute and action cues simultaneously.

Problem

Supervised STVG requires expensive frame-level annotations, and prior zero-shot methods rely on contrastive models (e.g., CLIP) that align only global image-text representations and thus localize poorly. Existing MLLM-based grounding approaches require task-specific fine-tuning with grounding datasets, risking catastrophic forgetting and limited generalization. Crucially, even when MLLMs possess latent grounding ability via special tokens, these tokens fail on complex queries because they cannot simultaneously integrate both spatial attribute cues and temporal action cues.

Method

Grounding Token Identification (GTI): Among all special tokens appended after the instruction prompt (e.g., <_A>, <IST>, <.>), the token with the highest visual activation (maximum text-to-visual attention inside the ground-truth region vs. outside) is selected per test sample as the "grounding token." This token's attention map over visual tokens drives bounding-box scoring.

Decomposed Spatio-Temporal Highlighting (DSTH): The original language query Q is decomposed (via GPT-4o in-context prompting) into an attribute sub-query Qs (for spatial inference) and an action sub-query Qt (for temporal inference). Each sub-query is rephrased as a yes/no question ("Is there [attribute] in this video?"). A learnable visual prompt variable (Vs or Vt, same shape as visual tokens) is added to the visual token stream. The Logit-Guided Re-Attention (LRA) module optimizes these prompts at test time by minimizing L = 1 - exp(logit_yes - logit_no), sharpening attention on the relevant spatial or temporal regions without modifying frozen LLM weights.

Temporal-Augmented Assembling (TAS): Because attribute sub-queries are temporally static, spatial grounding should be consistent across temporally perturbed inputs. TAS reverses the frame order, optimizes a separate spatial prompt on the reversed sequence, then assembles attention maps from both forward and reversed inputs to improve spatial localization robustness.

Final STVG prediction joins the object track score Sobj (from spatial attention of the grounding token over tracked proposals) and the frame score Sframe (from temporal attention).

Key Contributions

  • Discovery and empirical quantification of MLLM "grounding tokens"—special tokens that dynamically and selectively attend to query-relevant visual regions, with grounding accuracy correlating with visual activation rank.
  • DSTH strategy: query decomposition into attribute/action sub-queries combined with test-time LRA optimization to steer MLLM attention toward the correct spatial/temporal cues without fine-tuning.
  • TAS strategy: frame-reversal augmentation with prompt assembly to measure and alleviate temporal inconsistency in spatial grounding.
  • Demonstration across four MLLMs (LLaVA-Next-Video-7B, Qwen2-VL-7B, ShareGPT4Video-8B, LLaVA-OneVision-7B) and scalability validation on InternVL3 and Qwen2.5-VL series.

Results

  • HCSTVG-v1 (LLaVA-OneVision-7B): 24.8 m_vIoU / 41.5 vIoU@0.3 / 16.3 vIoU@0.5 vs. E3M (19.1 / 29.4 / 10.6); +12.1% vIoU@0.3, +5.7% vIoU@0.5.
  • HCSTVG-v2 (LLaVA-OneVision-7B): 27.7 m_vIoU / 44.7 vIoU@0.3 / 19.5 vIoU@0.5 (E3M not reported on v2).
  • VidSTG Declarative (LLaVA-OneVision-7B): 18.0 m_vIoU / 29.8 vIoU@0.3 / 12.2 vIoU@0.5 vs. E3M (16.2 / 20.5 / 11.9).
  • Outperforms all zero-shot baselines (E3M, ReCLIP, RedCircle) across all benchmarks and all tested MLLMs.
  • Surpasses SOTA weakly-supervised CoSPaL on HCSTVG-v2 by +5.5% m_vIoU.
  • Achieves results comparable to some fully-supervised methods.
  • Scalability: on InternVL3-8B, vIoU@0.3 increases from 36.5 (baseline) to 45.3 (ours); Qwen2.5-VL-7B from 39.7 to 45.4.
  • Ablation confirms GTI, DSTH (spatial + temporal), and TAS each contribute incrementally; contrasting yes/no logits outperforms entropy minimization.

Limitations

  • High computational cost from per-sample test-time optimization makes the method slow and unsuitable for long videos.
  • Depends on an external detector (Grounding DINO) and tracker (SAM2) for tubelet proposal generation; tracker quality directly affects STVG performance.
  • Query decomposition requires GPT-4o as an external LLM, adding API cost and latency.
  • Performance degrades for smaller MLLM scales (e.g., 3B–4B models) due to weaker multimodal alignment.
  • Evaluated only on three STVG benchmarks; generalization to more diverse video domains is unverified.

Relevance to Vision-Language Models

This work provides direct mechanistic insight into how MLLMs process grounding queries: the special structural tokens appended after user instructions act as implicit spatial anchors, and their visual activation correlates with localization accuracy. This finding is relevant to any work studying the emergent visual localization capabilities of autoregressive MLLMs without explicit grounding supervision. The LRA module—optimizing learnable visual token perturbations via yes/no token logit contrast at test time—offers a lightweight, fine-tuning-free adaptation paradigm applicable beyond STVG to other visual reasoning tasks. The paper also highlights that compositional query understanding (attribute vs. action) remains a systematic failure mode of current MLLMs, pointing to a gap that future VLM architectures and instruction-tuning curricula should address.