Skip to content

Unified Interactive Multimodal Moment Retrieval via Cascaded Embedding-Reranking and Temporal-Aware Score Fusion

🕒 Published (v1): 2025-12-15 02:50 UTC · Source: Arxiv · Venue: AAAI · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper presents a unified multimodal video moment retrieval system combining cascaded dual-embedding retrieval (BEiT-3 + SigLIP → BLIP-2 reranking), exponential-decay temporal scoring via beam search, and GPT-4o-driven query decomposition with adaptive modality fusion. The system eliminates manual modality selection while handling ambiguous natural-language queries over visual, OCR, and ASR channels. It scored 76.4/88 (86.8%) at AI Challenge 2025.

Problem

Existing multimodal moment retrieval systems suffer three compounding failures: (1) fixed-weight fusion degrades under cross-modal noise and ambiguous queries; (2) temporal modeling either uses rigid fixed windows or attention without explicit gap penalties, producing incoherent frame sequences; (3) systems require users to manually specify retrieval modalities, reducing usability for real-world queries.

Method

Offline indexing: TransNetV2 shot segmentation extracts 3 keyframes/shot; visual embeddings from BEiT-3 and SigLIP stored in Qdrant; OCR via Gemini 2.0 Flash; ASR via Whisper Large-v3.

Cascaded retrieval: BEiT-3 + SigLIP dual encoders retrieve top-100 candidates via Score-Reflected Reciprocal Rank Fusion (SRRF); BLIP-2 Image-Text Matching cross-encoder reranks for precision.

Temporal search: Beam search (width B=8) constructs K-event sequences from M candidates per event in O(B×K×M). Exponential decay λᵢ = e^(−α·Δtᵢ) with α=0.01 penalizes large inter-event gaps. Sequence score is additive (SSⱼ = Σ sᵢ·e^(−α·Δtᵢ)); final reranking applies multiplicative gating with BLIP-2 score bᵢ: Sᵢ^(final) = sᵢ·λᵢ·bᵢ.

Agent-guided fusion: GPT-4o decomposes the query into visual/OCR/ASR sub-queries with predicted weights wₘ; GPT-4o also generates N=4 query expansion variants. Min-max normalization aligns cross-modal scores; final score S(f) = Σ wₘ·sₘ^norm(f).

Key Contributions

  • Cascaded dual-encoder (BEiT-3 + SigLIP) + BLIP-2 reranking pipeline balancing recall and precision at scale
  • Exponential decay temporal scoring with beam search for coherent multi-event sequence retrieval
  • GPT-4o agent for automatic query decomposition, modality routing, weight prediction, and query expansion — no manual modality specification required
  • Score-Reflected Reciprocal Rank Fusion (SRRF) preserving original similarity magnitudes over rank-only RRF
  • Two-stage scoring architecture: additive aggregation for exploration, multiplicative gating for final validation

Results

  • AI Challenge 2025: 76.4/88 overall (86.8%), advancing to the final round
  • Round 1: 19.8/23 (86.1%)
  • Round 2: 26.6/30 (88.6%)
  • Round 3: 30/35 (85.7%)
  • Dataset: ~1,500 videos, >200 GB; tasks: KIS (72 queries), VQA (9), TRAKE (7)
  • Qualitative: cascaded reranking recovers correct ground-truth frames missed by single-stage retrieval; agent-guided OCR weighting (w_ocr≈0.7 vs w_vis≈0.4) resolves visually ambiguous scenes; temporal search with λ=0.01 produces 5.1s coherent sequences (score 0.9234) vs fragmented 34.1s alternatives
  • No ablation table with numeric baselines provided; comparisons are qualitative only

Limitations

  • No quantitative ablation study; contributions are validated qualitatively rather than with controlled metrics (e.g., R@K, mAP)
  • BLIP-2 reranking is O(n) per query over the candidate set, limiting scalability beyond top-100
  • Evaluation confined to a single competition dataset; no validation on standard benchmarks (TRECVID, QVHighlights, VBS)
  • Exponential decay hyperparameter α requires manual tuning and may not generalize across domains with different temporal scales
  • GPT-4o latency and cost for query decomposition and expansion not characterized; unsuitable for strict real-time constraints
  • Query expansion always generates English variants, which may disadvantage non-English corpora

Relevance to Agentic AI / LLM Agents

The paper is a concrete example of an LLM acting as a system-level orchestrator: GPT-4o does not merely embed or generate text but reasons about query intent, decomposes it into typed sub-tasks, assigns weights to downstream modules, and drives adaptive fusion — a narrow but real instance of agentic tool routing. This pattern — LLM as planner/router over a heterogeneous retrieval pipeline — is directly applicable to multi-modal RAG agents and tool-use architectures. The approach of prompt-driven modality routing without a trained routing model demonstrates how modern LLMs reduce the need for specialized classifiers in agentic pipelines. It also surfaces a practical challenge for agent designers: how to dynamically weight and fuse outputs from independent retrieval tools when each tool returns scores on incompatible scales.