Skip to content

TimeLens2: Generalist Video Temporal Grounding with Multimodal LLMs

🕒 Published (v1): 2026-07-19 22:04 UTC · Source: Arxiv · link

Why this paper was selected

Generalist video temporal grounding with MLLMs; fills key capability gap in video VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TimeLens2 is a generalist video temporal grounding framework that trains multimodal LLMs to predict sets of evidence intervals across diverse video lengths, domains, query types, and viewpoints. It addresses two core mismatches—unreliable multi-span supervision and geometry-blind RL rewards—via a staged data pipeline (TimeLens2-93K) and a matching-free temporal Wasserstein reward. The 4B model surpasses open-source models up to 397B parameters across seven benchmarks.

Problem

Existing video MLLMs can describe video content but fail to localize when evidence occurs. Two structural gaps block generalist temporal grounding: (1) long-video annotation is brittle because one-pass global labeling misses repeated evidence and produces imprecise boundaries; (2) RL rewards based on temporal IoU assign zero to all disjoint predictions, collapsing near-misses and distant errors to the same signal, while one-to-one segment matching (used by alternatives like MUSEG) is fragile under fragmentation or unequal cardinality.

Method

TimeLens2-93K data pipeline (6 stages): 1. Stratified sampling of 34,867 YouTube videos across five duration bands and 15 domains. 2. Hierarchical time-stamped captioning with Qwen3-VL-235B using PySceneDetect clips (20–60 s). 3. Segment-aware query synthesis via Kimi-K2.5, jointly producing declarative queries and coarse caption-derived proposals. 4. Dual-agent local grounding: Qwen3-VL-30B-A3B and TimeLens-8B independently relocalize each proposal from the raw video, returning interval sets or ∅. 5. Cross-agent consensus gate (\(\text{IoU}_\text{set}(A,B) > 0.9\)) plus semantic verification (Qwen3-VL-Embedding cosine similarity \(> 0.5\) between query and target clip). 6. Boundary-focused local refinement using ±3 s neighborhood search; adjacent gaps ≤ 1 s are merged.

Training follows a two-stage recipe on Qwen3-VL-2/4/8B: - SFT on long-context (100K-token) examples from TimeLens2-93K + TimeLens-100K + Ego4D-NLQ with format diversity (JSON, natural language, key–value, range styles). - GRPO with hard-sample mining (eight off-policy rollouts; low-tIoU examples up-weighted) and a composite reward: $\(R(\hat{Y}, Y) = R_{\text{tIoU}}(\hat{Y}, Y) + R_{\text{TW}}(\hat{Y}, Y) - \mathbf{1}_{\text{invalid}}\)$

The temporal Wasserstein reward lifts each interval set to a uniform distribution over its merged support \(\mu_A(t) = \mathbf{1}\{t \in \text{merge}(A)\} / \ell(A)\) and computes the exact 1D \(W_1\) distance: $\(W(\hat{Y}, Y) = \int_{\mathbb{R}} |F_{\hat{Y}}(t) - F_Y(t)|\, dt\)$ normalized by target duration: \(R_{\text{TW}} = \exp\!\bigl(-W / (|\text{merge}(Y)| + \epsilon)\bigr)\).

Key Contributions

  • TimeLens2-93K: A 93K-instance corpus with 23,793 videos (including 12,091 multi-span instances) built through staged evidence verification (consensus + semantic alignment + boundary refinement).
  • Temporal Wasserstein reward: Matching-free, fragmentation-invariant \(W_1\)-based RL signal providing dense geometric feedback for disjoint and unequal-cardinality interval sets.
  • Generalist grounding models at 2B, 4B, 8B scale achieving state-of-the-art across seven benchmarks spanning short, long, multi-span, question-form, and egocentric grounding.

Results

  • TimeLens2-2B: 44.5 average mIoU, outperforms all size-matched baselines on all 7 benchmarks; +14.2 mIoU over Qwen3-VL-2B backbone; +2.4 over TimeLens-8B.
  • TimeLens2-4B: 47.7 average mIoU; surpasses Qwen3.5-397B-A17B on every benchmark by 7.5 points on average; best open-source on 6/7 benchmarks.
  • TimeLens2-8B: 48.0 average mIoU; surpasses all prior methods in R1@0.5 on all 7 benchmarks; trails Gemini 2.5 Pro on QVHighlights by only 0.2 mIoU.
  • Backbone improvements: +14.2 / +13.0 / +18.1 mIoU over Qwen3-VL-2B/4B/8B respectively.
  • Temporal Wasserstein restores informative reward signal for 75.8% of GRPO groups that otherwise had all-zero tIoU.
  • Declarative-only TimeLens2-93K raises question-form grounding on MomentSeeker from 15.3 → 25.8 mIoU.

Limitations

  • Visual encoder is frozen throughout both training stages; fine-tuning it might yield further boundary precision.
  • Annotation quality depends on intermediate model outputs (Qwen3-VL-235B, TimeLens-8B); errors in the annotation chain may propagate despite verification.
  • Videos are sampled at 2 fps and capped at 512 frames, potentially losing fine-grained temporal detail for fast-paced events.
  • The 1-second annotation resolution in dual-agent grounding may introduce systematic quantization error in boundary labels.
  • Evaluation is limited to mIoU and R1@0.5/0.3/0.7; no user-study or downstream application validation is reported.

Relevance to Vision-Language Models

TimeLens2 directly extends the capability of video MLLMs by adding verifiable temporal grounding—transforming fluent video descriptions into traceable, interval-cited outputs analogous to document citations. The work demonstrates that a compact 4B MLLM trained with high-quality set-valued supervision and geometry-aware RL can outperform 397B-parameter models, highlighting data quality and reward design as critical bottlenecks rather than scale alone. The temporal Wasserstein reward is broadly applicable to any VLM trained with GRPO-style RL on tasks requiring set prediction under unequal cardinality. The instruction- and format-diversity training strategy offers a reusable recipe for improving cross-benchmark generalization in video VLMs.