Skip to content

Time-R1: Post-Training Large Vision Language Model for Temporal Video Grounding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Time-R1 is a reinforcement learning post-training framework that improves Large Vision-Language Models (LVLMs) on Temporal Video Grounding (TVG) — localizing video segments from natural language queries. Using only 2.5K training samples, it outperforms SFT-based LVLMs trained on hundreds of thousands of examples and achieves state-of-the-art TVG performance while preserving general video QA capability.

Problem

LVLMs with 7B+ parameters underperform smaller feature-based TVG models (e.g., a 9M-parameter model) despite pretraining on 100× more data. The root cause is identified as over-penalization of near-correct predictions during SFT: the autoregressive loss assigns high loss even to temporally close but non-exact timestamp predictions, causing overfitting and poor generalization. Existing workarounds (new timestamp tokens, regression heads) sacrifice pretrained language capabilities.

Method

Time-R1 applies Generalized Reinforcement Policy Optimization (GRPO) to post-train LVLMs for TVG with a composite verifiable reward:

  1. Timestamp-aware IoU reward (tIoU): Extends standard IoU by multiplicatively penalizing per-endpoint deviation relative to video duration — tIoU = IoU × (1 − |ts−ts'|/t) × (1 − |te−te'|/t) — preventing inflated rewards for wide-span predictions.
  2. Reasoning template reward (rform): Binary reward for generating well-formed <think>…</think><answer>ts to te</answer> chain-of-thought output.

The LVLM generates G candidate responses; GRPO maximizes the group-normalized weighted reward minus a KL-divergence penalty. The visual encoder is frozen; only the LLM is updated.

TimeRFT (RL-friendly fine-tuning strategy): - Curates 2.5K samples from 339K by Gaussian filtering over per-sample IoU centered at 0.3 (moderate difficulty — not trivially easy or reward-starved hard). - Applies multi-epoch training with per-epoch dynamic hard sampling: after each epoch, samples with IoU > 0.7 are dropped. - For 3B models: cold-start SFT on small CoT-formatted data (sequential timestamped captions) before RL to suppress hallucinated reasoning.

TVGBench: A curated 800-instance benchmark balanced across 5 source datasets, 11 query semantic types (human/object/environment), video durations, and query center positions.

Key Contributions

  • Time-R1: GRPO-based post-training framework for TVG with tIoU + template composite reward and explicit chain-of-thought reasoning prior to timestamp prediction.
  • TimeRFT: Data curation (Gaussian IoU filtering, 2.5K samples from 339K) and training strategy (multi-epoch + dynamic hard sampling + cold-start CoT SFT) for data-efficient RL fine-tuning.
  • TVGBench: Compact (800 instances), balanced, LVLM-appropriate benchmark spanning 5 datasets and 11 query semantic categories with manual semantic annotation.
  • Demonstration that RL post-training generalizes across multiple LVLM families (Qwen2.5-VL, MiMo-VL, InternVL3) and sizes (2B–8B).

Results

  • Charades-STA (zero-shot): Time-R1 R1@0.3 = 78.1, R1@0.5 = 60.8, R1@0.7 = 35.3 — surpasses VideoChat-Flash (74.5 / 53.1 / 27.6) and VideoMind (72.5 / 59.1 / 31.2).
  • Charades-STA (fine-tuned, Time-R1*): R1@0.5 = 72.2, R1@0.7 = 50.1 — outperforms TimeSuite* (67.1 / 43.0) and VideoChat-TPO* (65.0 / 40.7) with far fewer training samples.
  • ActivityNet (zero-shot): R1@0.3 = 58.6, R1@0.5 = 39.0 — surpasses HawkEye (49.1 / 27.8) and VideoMind (48.4 / 30.3).
  • ActivityNet (fine-tuned): R1@0.5 = 55.6 — exceeds SSRN* VLP (54.5) and TRACE SFT (37.7).
  • TVGBench: R1@0.3 = 41.8, beats TRACE (37.0) and Gemini-2.5-Pro (39.1).
  • General video QA (VideoMME long): RL improves from 53.0 to 54.2; SFT-LoRA degrades to 51.7.
  • ActivityNet mIoU: RL reaches 29.2 vs. SFT-LoRA 25.9 (on 339K samples) vs. base 16.3.
  • Cross-model gains (TVGBench R1@0.7): MiMo-VL-7B 6.6→15.1; Qwen2.5-VL-3B 3.8→10.5; InternVL3-8B 3.4→9.2.

Limitations

  • Cannot process ultra-long videos (explicit in the paper); long-horizon context is unaddressed.
  • TVGBench is limited to 800 instances; HAS (Human Action Simple) category is overrepresented due to inherent dataset bias.
  • Frame sampling at 2 FPS with ~96×96 resolution per frame limits fine-grained temporal resolution in longer videos.
  • Cold-start SFT data (CoT formatting) is an additional requirement for smaller models to achieve stable training.

Relevance to Vision-Language Models

This paper directly addresses a known failure mode of LVLMs — their underperformance on structured localization tasks despite large-scale pretraining — by replacing SFT with RLVR, following the DeepSeek-R1 paradigm now being applied across modalities. It demonstrates that IoU-based verifiable rewards can serve as a strong training signal analogous to exact-match rewards in math/code, extending RLVR from image and short-video reasoning into long-form video understanding. The finding that 2.5K RL samples outperform 339K SFT samples is highly relevant to data-efficient LVLM adaptation. TVGBench also offers a reusable diagnostic tool for the community evaluating temporal reasoning in LVLMs.