Skip to content

Mitigating Modality and Language-Style Gaps for Zero-Shot Video Moment Retrieval

🕒 Published (v1): 2026-07-21 12:16 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Why this paper was selected

ECCV 2026; zero-shot VMR resolving modality and language-style alignment gaps

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Self-SiMS addresses the modality and language-style gaps that plague zero-shot video moment retrieval (ZMR) by replacing query-dependent similarity signals with query-independent intra-video temporal self-similarity for span generation and scoring. A contrastive kernel detects temporal boundaries from frame/caption self-similarity maps, and an MLLM Yes/No re-ranking stage provides query-aware final selection. The method achieves state-of-the-art on QVHighlights, Charades-STA, ActivityNet-Captions, and TVR without any training.

Problem

Existing ZMR methods rely on query–frame or query–caption similarity to generate and score moment proposals. Query–frame similarity suffers from a modality gap (text vs. visual feature spaces are misaligned), producing low, flat signals even inside ground-truth moments. Query–caption similarity partially closes that gap but introduces a language-style gap: MLLM-generated captions and human-written queries inconsistently describe the same scene, yielding noisy, unstable similarity trajectories. Both failure modes cause unreliable span proposals and mis-scoring.

Method

Self-Similarity-based Moment Proposal and Scoring (Self-SiMS) operates in three stages:

  1. Temporal Self-Similarity Maps. Frame features \(\hat{F}^f\) and MLLM caption features \(\hat{F}^c\) (both \(\ell_2\)-normalized) produce \(M^f = \hat{F}^f(\hat{F}^f)^\top\) and \(M^c = \hat{F}^c(\hat{F}^c)^\top\), merged as \(M = \tfrac{1}{2}(M^f + M^c)\). A contrastive kernel \(K \in \mathbb{R}^{N_K \times N_K}\) (positive in top-left/bottom-right quadrants, negative in off-diagonal quadrants) is convolved with diagonal patches of \(M\) to produce per-frame boundary scores \(b_i = P_i \odot K\). Dynamic instance-wise thresholds (scaled by intra-video feature variance) extract local maxima as boundaries, generating candidate spans \(\mathcal{E}\).

  2. Self-Similarity-Aware Span Scoring. Each span \(E_n\) receives:

  3. Query-Matching Span Score \(S^Q_n\): mean of the top-\(k_S\) query–caption cosine similarities within the span (robust to noisy frames).
  4. Self-Matching Span Score \(S^S_n\): mean TSM similarity between the key frame \(\kappa_n = \arg\max_{i \in E_n} S^f_i\) and all other span frames, penalizing temporally incoherent spans.

  5. Combined: \(S_n = (1-\alpha)S^Q_n + \alpha S^S_n\).

  6. Query-Aware MLLM Re-ranking. Top-\(k_C\) spans are re-ranked using LLaMA-3.2-11B-Vision-Instruct with a Yes/No prompt per sampled representative frame; the softmax Yes-probability averaged over top-\(k_R\) frames gives \(S^R_{(m)}\). Final score: \(S^*_{(m)} = (1-\beta)S_{(m)} + \beta S^R_{(m)}\).

Key Contributions

  • First explicit identification and joint treatment of both modality and language-style gaps in training-free ZMR.
  • First use of temporal self-similarity (via a contrastive boundary kernel) for candidate span generation in zero-shot VMR.
  • Self-Matching Span Score: uses the TSM to enforce intra-span temporal coherence during scoring, reducing reliance on noisy query–caption alignment.
  • Query-aware MLLM Yes/No re-ranking that directly evaluates query–frame relevance without independent embedding comparison.
  • SOTA across four ZMR benchmarks (QVHighlights, Charades-STA, ActivityNet-Captions, TVR) with no training.

Results

QVHighlights test set (ZS vs. best ZS baseline Moment-GPT): - R1@0.5: 59.7 vs. 58.3 (+1.4) - R1@0.7: 42.2 vs. 37.7 (+4.5) - mAP@0.5: 59.2 vs. 55.1 (+4.1) - mAP@avg: 38.3 vs. 35.0 (+3.3)

Charades-STA (ZS): - R1@0.5: 39.7 vs. 38.4 (Moment-GPT); mIoU: 41.9 vs. 38.4

ActivityNet-Captions (ZS): - R1@0.3: 49.9 vs. 48.1 (Moment-GPT); mIoU: 34.7 vs. 30.8

TVR (ZS): - R1@0.3: 26.6 vs. 14.4 (TFVTG‡, best ZS prior); mIoU: 20.2 vs. 18.6

  • Self-SiMS outperforms all ZS methods on all four benchmarks and approaches or matches weakly-supervised methods (e.g., CPL R1@0.5 = 38.1 on Charades-STA).

Limitations

  • Relies on MLLM-generated captions (LLaMA-3.2-11B-Vision-Instruct) for both TSM construction and re-ranking; MLLM quality and inference cost directly impact performance.
  • Low-fps sampling (0.5–1 fps) may miss fast or brief events; boundary detection quality degrades for very short moments.
  • The variability-aware dynamic thresholds and hyperparameters (\(\alpha\), \(\beta\), \(k_S\), \(k_C\), \(k_R\), \(N_R\)) are dataset-tuned; generalization to radically different video domains is not demonstrated.
  • Only Yes/No binary prompting is used for re-ranking; richer MLLM reasoning is not exploited.
  • Paper is truncated before reporting ablation details and full ActivityNet/TVR analysis.

Relevance to Vision-Language Models

Self-SiMS directly probes a fundamental weakness of VLMs in cross-modal retrieval: the embedding space misalignment between text and visual modalities causes query–frame similarity to fail even inside ground-truth segments. By bypassing VLM query-to-video similarity for span generation altogether—relying instead on intra-video structure—this work demonstrates a practical zero-shot regime where VLMs (CLIP, BLIP-2) are used only as feature extractors, while an MLLM (LLaMA-3.2-Vision) provides query-aware reasoning as a late-stage verifier. This suggests a design principle for VLM-based retrieval: decouple proposal generation (where modality gaps are most harmful) from query-aware selection (where MLLMs excel), which has implications beyond VMR for any temporal or spatial grounding task using frozen VLMs.