Universal Video Temporal Grounding with Generative Multi-modal Large Language Models¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

UniTime is a universal video temporal grounding model that adapts generative MLLMs (specifically Qwen2-VL-7B) to localize temporal moments in videos by interleaving explicit timestamp tokens with visual tokens. It addresses the cross-domain and cross-duration limitations of existing grounding models via adaptive frame scaling and coarse-to-fine multi-stage inference. UniTime outperforms state-of-the-art methods across five temporal grounding benchmarks in both zero-shot and fine-tuned settings.

Problem¶

Existing video temporal grounding (VTG) methods are either (1) discriminative/DETR-based models that are dataset-specific and lack strong language understanding, or (2) MLLM-based models bottlenecked by short context windows and GPU memory, preventing them from handling long videos. No prior model robustly generalizes across diverse video durations (seconds to hours), viewpoints (egocentric/exocentric), and query types (descriptions, questions, procedural).

Method¶

UniTime fine-tunes Qwen2-VL-7B via LoRA on temporal grounding data using three core mechanisms:

Adaptive frame scaling: Dynamically allocates a fixed total token budget (N_total) across frames based on video duration — high spatial resolution for short videos (frame resize), low resolution with bilinear token compression for long videos. Thresholds N_f^short=128 and N_f^long=1024 govern the regime.
Timestamp-interleaved sequence construction: Timestamps are encoded as free text (e.g., "timestamp: 15.0 seconds") and interleaved before visual tokens at either frame or segment granularity. The LLM retrieves boundary timestamps directly from these inserted tokens (e.g., outputs "From sk seconds to ek seconds"), avoiding dense position-encoding regression.
Multi-stage coarse-to-fine inference: For long videos exceeding N_f^long frames, the video is partitioned into clips; coarse segment retrieval is performed first, then fine-grained grounding is applied within retrieved segments recursively. Training uses video-centric batching — all queries for a video are concatenated in a single pass with masked cross-attention — to avoid redundant video re-encoding.

Training data spans seven pre-training datasets (NaQ, DiDeMo, COIN, Momentor, YouCook2, etc.) and five benchmark training sets, covering egocentric, exocentric, and mixed views across domains.

Key Contributions¶

Timestamp-interleaved sequence design that is model-agnostic and alignment-free (tokens live in language space, no learned temporal embeddings needed)
Adaptive frame scaling that dynamically trades spatial resolution for temporal coverage across variable-duration videos
Coarse-to-fine multi-stage inference enabling generalization from training on ~6-minute videos to inference on 20+ minute videos
Video-centric training paradigm eliminating redundant video loading/encoding across multiple queries per video
State-of-the-art performance across five benchmarks in zero-shot, universal (single model), and dataset-specific settings

Results¶

Dataset-specific (UniTime-SP) vs. prior best: - Ego4D-NLQ: R1@0.3 +8.81 pp, mIoU +9.10 pp over LD-DETR - TaCoS: R1@0.5 +10.44 pp over SnAG - Charades-STA R1@0.5: 75.27 vs. 70.20 (Mr.BLIP), +5.07 pp - QVHighlights R1@0.5: 77.76 vs. 74.77 (TimeSuite), +1.95 pp

Zero-shot (UniTime-Zero) vs. prior best: - Ego4D-NLQ mIoU: +25.47 pp over TimeMarker - TaCoS mIoU: +15.44 pp over TimeMarker - Charades-STA R1@0.5: 59.09 vs. 51.90 (TimeMarker), +7.19 pp

Closed-source comparison: UniTime-Full outperforms Gemini-2.5-Pro and GPT-4o on Ego4D-NLQ; competitive with Seed1.5-VL on Charades-STA, with more balanced short/long video performance than any single closed-source model.

Downstream VideoQA (as retriever for Qwen2-VL-7B): - MLVU: 66.50 vs. 62.56 (UniVTG baseline) - LongVideoBench: 56.47 vs. 54.82 (uniform sampling) - QaEgo4D IoP: 25.29 vs. 9.80 (UniVTG) - CG-Bench IoP: 16.64 vs. 5.06 (TimeSuite)

Robustness: UniTime-SP achieves 80.20 ratio on time-shift perturbation vs. 66.77 (Mr.BLIP), 66.26 (TimeSuite), demonstrating lower distributional bias sensitivity.

Flexibility: Method transfers to Qwen2-VL-2B, InternVL2.5-2B/8B with consistent large gains (e.g., Ego4D-NLQ R1@0.3: +10.50 on 2B base, +24.79 on 7B base from near-zero baselines).

Limitations¶

Coarse-to-fine inference adds latency for long videos — multi-stage recursive retrieval is not analyzed for latency tradeoffs
Frame rate is fixed at 2 fps; very fast-action videos may lose critical frames regardless of spatial resolution
Segment length (32 frames) is a fixed hyperparameter; longer segments improve retrieval but hurt fine-grained localization, creating an inherent accuracy-recall tradeoff
LoRA fine-tuning with rank=8 may leave significant MLLM capacity untapped; full fine-tuning is not evaluated
Generalization to >20-minute videos at inference is demonstrated but limited benchmarking exists for hour-long content

Relevance to Vision-Language Models¶

UniTime directly extends the capabilities of generative MLLMs (Qwen2-VL, InternVL) to the temporally-grounded video domain, showing that explicit textual timestamp interleaving is a lightweight, plug-and-play mechanism that avoids the alignment complexity of learned temporal embeddings. This is a concrete recipe for adapting any dynamic-resolution MLLM to temporal reasoning tasks without architectural changes — directly relevant to the ongoing line of work on grounding and spatial-temporal understanding in VLMs. The result that a 7B-parameter open model trained with LoRA surpasses GPT-4o and Gemini-2.5-Pro on long-video grounding challenges the assumption that proprietary scale is necessary for fine-grained temporal localization. The retrieval-then-reasoning paradigm validated here reinforces temporal grounding as a key capability gap for VLMs operating on video.