Skip to content

Empower Words: DualGround for Structured Phrase and Sentence-Level Temporal Grounding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DualGround is a dual-branch Video Temporal Grounding (VTG) architecture that disentangles sentence-level ([EOS] token) and phrase-level (clustered word tokens) semantics to overcome the over-reliance on global [EOS] representations prevalent in prior models. It achieves state-of-the-art moment retrieval and highlight detection on QVHighlights and Charades-STA by routing each semantic level through a dedicated cross-modal interaction pathway and fusing them for final prediction.

Problem

Existing VTG models using pretrained VLMs (CLIP, InternVideo2) treat all text tokens uniformly in cross-modal attention, causing attention to collapse onto the [EOS] summary token and neglect fine-grained word-level cues. Controlled ablations show models achieve comparable or better performance with [EOS]-only input versus full token sequences, confirming that word tokens are systematically underutilized despite carrying visually discriminative information (e.g., "red jacket").

Method

DualGround processes text through two parallel paths that are fused additively (V_s + V_p) before multi-scale temporal decoding:

Sentence-level path: Uses only the [EOS] token as the key in Adaptive Cross Attention (ACA), augmented with learnable dummy tokens that act as attention sinks for off-topic clips, enabling sharp global alignment without interference from noisy word tokens.

Phrase-level path: 1. Recurrent Phrase Generation (RPG): An N-step recurrent module composes N phrase embeddings by softly aggregating word tokens via scaled dot-product attention, conditioned on [EOS] and prior phrase embeddings, with positional injection to encourage adjacency grouping. 2. Slot Attention refinement: Refines phrase embeddings by treating them as latent slots that iteratively aggregate word-level key-value pairs, disentangling overlapping meanings. 3. Phrase-Clip Context: Dense N×T context embeddings computed via Hadamard product of projected phrase and video features, followed by temporal self-attention. 4. Phrase-Guided Aggregation: A learnable P_[EOS] token summarizes phrase semantics and computes importance weights for aggregating across phrases per clip.

Training losses include standard MR (focal + L1) and HD (ranking + contrastive) losses, plus phrase-level supervision: Distinct Query Attention (DQA) loss (Frobenius norm on inter-phrase attention to enforce orthogonality) and EOS Reconstruction loss (InfoNCE aligning P_[EOS] with e_[EOS]).

Decoding follows FlashVTG's multi-scale temporal pyramid via 1D convolutions, bypassing DETR-style fixed query decoding.

Key Contributions

  • Empirical diagnosis of [EOS]-over-reliance bias in existing VTG models via controlled token ablations and attention visualization.
  • DualGround dual-branch architecture that structurally disentangles global ([EOS]) and local (phrase) semantic paths for cross-modal attention.
  • Recurrent Phrase Generator (RPG) + Slot Attention refinement pipeline for constructing semantically coherent, disentangled phrase representations.
  • Phrase-level training objectives (DQA loss + EOS Reconstruction loss) ensuring diversity and global consistency of phrase representations.
  • State-of-the-art on QVHighlights (test and val) and Charades-STA under both CLIP+SlowFast and InternVideo2 backbones.

Results

QVHighlights Test (SF+C backbone): - DualGround: R1@0.5=68.20, R1@0.7=51.72, mAP=49.02 vs. FlashVTG (best prior): R1@0.5=66.08, R1@0.7=50.00, mAP=48.70

QVHighlights Test (IV2 backbone): - DualGround: R1@0.5=71.87, R1@0.7=56.94, mAP=52.73 vs. FlashVTG: R1@0.5=70.69, R1@0.7=53.96, mAP=52.00

QVHighlights Val (IV2 backbone): - DualGround: R1@0.5=73.48, R1@0.7=58.97, mAP=53.26 vs. FlashVTG: 71.48, 56.06, 52.61

Charades-STA (IV2): - DualGround: R1@0.5=70.67, R1@0.7=50.33 vs. FlashVTG: 70.32, 49.87

Long-query robustness (QVHighlights val, >20 words, mAP): - DualGround: 48.92 vs. FlashVTG: 43.46, TR-DETR: 35.33

Limitations

  • Fixed number of phrases N per query requires manual tuning per dataset; cannot adapt to variable-length or structurally diverse queries.
  • Audio modality is not incorporated, limiting performance in audio-visual grounding scenarios.
  • Phrase-level path can degrade Highlight Detection metrics when DQA or EOS Reconstruction losses are omitted, indicating sensitivity to hyperparameter balance.
  • Evaluated only on standard VTG benchmarks; generalization to open-world or zero-shot settings is unverified.

Relevance to Vision-Language Models

DualGround directly interrogates a structural limitation in how pretrained VLMs (CLIP, InternVideo2) expose text representations to downstream cross-modal tasks: the [EOS] pooling design, while effective for contrastive pretraining, creates an inductive bias that suppresses word-level semantics in fine-grained alignment tasks. The paper's phrase-level architecture is a lightweight adapter that recovers sub-sentence structure without fine-tuning the backbone VLM, demonstrating that representation disentanglement at the consumer layer can unlock capabilities latent in frozen VLM encoders. This connects to the broader line of work on probing and exploiting the internal token structure of VLMs (e.g., keyword-aware attention, holistic query understanding) and has implications for any VLM-based grounding task (referring expression comprehension, visual question answering) where global pooling obscures locally discriminative tokens.