Skip to content

Scaling RL to Long Videos

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LongVILA-R1 is a full-stack framework that scales reinforcement learning-based reasoning in VLMs to long videos (up to 8,192 frames). It combines a new 104K-sample long-video reasoning dataset (LongVideo-Reason), a two-stage CoT-SFT → GRPO training pipeline, and Multi-modal Reinforcement Sequence Parallelism (MR-SP) to overcome the memory and compute bottlenecks of RL on long visual sequences. LongVILA-R1-7B achieves 65.1%/71.1% on VideoMME (w/o/w subtitles) and 72.0% on the new LongVideo-Reason-eval benchmark, outperforming Gemini-1.5-Pro on the latter.

Problem

Existing VLM reasoning systems (e.g., Video-R1) operate on at most 16 frames, making them ill-suited for long video understanding that demands temporal, spatial, goal-oriented, and narrative reasoning across minutes or hours of footage. Two gaps block progress: (1) no large-scale, high-quality long-video reasoning dataset with chain-of-thought annotations exists; and (2) RL frameworks for VLMs are not designed for the massive token volumes of long videos—rollout sampling and LLM prefilling become prohibitively expensive or cause GPU OOM.

Method

Dataset construction (LongVideo-Reason): 18K long videos are segmented into ~10-second clips; NVILA-8B captions each clip; DeepSeek-R1-671B generates diverse Question-Reasoning-Answer triples (temporal, spatial, goal/purpose, plot/narrative) from the concatenated captions. Spatial QAs additionally use VILA-HD bounding boxes. A test-time scaling filter (10 inference passes per question) removes too-easy and too-hard samples, retaining 36K for CoT-SFT warm-up and 68K for RL; 102K open-source samples are appended for RL scaling.

Training pipeline: - Stage 1 — Long CoT-SFT: Fine-tunes LongVILA on 36K samples with <think>…</think><answer>…</answer> format using the existing MM-SP system. - Stage 2 — GRPO RL: Standard GRPO with group size G=8 and rule-based format/accuracy rewards on 68K+102K samples.

MR-SP (Multi-modal Reinforcement Sequence Parallelism): - Rollout stage: Video frames are sharded across GPUs, each GPU encodes its frame slice with its own vision tower, and an all-gather operation assembles the full embedding once. These cached embeddings are reused across all G rollouts within a training step, eliminating redundant re-encoding. - Prefilling stage: The gathered embeddings are padded to uniform length and sharded across GPUs via sequence parallelism so each GPU handles only a token slice during attention/logit computation for both policy and reference models.

Key Contributions

  • LongVideo-Reason dataset: 104K long-video QA pairs with full CoT annotations across four reasoning categories; accompanied by LongVideo-Reason-eval, a manually curated 1K-sample benchmark.
  • MR-SP training system: Sequence-parallel video encoding with embedding reuse + SP-based prefilling, achieving 2.1× speedup at 512 frames and enabling 1,024-frame RL without OOM on a single 8×A100 node.
  • Two-stage training recipe: CoT-SFT warm-up followed by GRPO RL, with ablations showing both stages are necessary (skipping CoT-SFT drops accuracy; using inferior datasets degrades performance).
  • LongVILA-R1-7B model: Supports up to 8,192 frames with configurable FPS; open-sourced along with the training system supporting video/text/audio modalities and VILA/Qwen model families.

Results

  • VideoMME (512 frames, 7B scale): 65.1% w/o subtitles, 71.1% w/ subtitles — top among 7B-class open models; vs. LongVILA-7B baseline: +5.0/+6.0 points.
  • LongVideo-Reason-eval: 72.0% overall vs. Video-R1-7B (68.1%) and Gemini-1.5-Pro (69.3%).
  • ActivityNet-QA: 64.8% vs. LongVILA-7B 59.5%.
  • PerceptionTest: 68.9% vs. LongVILA-7B 58.1%.
  • VNBench: 75.5% vs. LongVILA-7B 63.0%.
  • MR-SP speedup: 2.1× at 512 frames over no-MR-SP baseline; enables 1,024-frame training that otherwise OOMs.
  • Frame scaling ablation (1.5B model): LongVILA-R1-1.5B improves monotonically from 55.9% at 16 frames to 64.3% at 512 frames, while the SFT-only baseline degrades after 256 frames.

Limitations

  • Single-node RL (8×A100) supports up to ~3,600 frames (hour-long video); further scaling to longer sequences or larger batches requires multi-node distributed training, which is more resource-intensive.
  • Dataset annotation pipeline relies on caption-level indirect supervision (no direct frame grounding); spatial QA quality is bounded by VILA-HD detection accuracy.
  • RL training cost is high (~80K H100 GPU hours for dataset generation alone); accessible only to well-resourced labs.
  • Audio modality is not integrated into the long-video reasoning pipeline; the framework mentions audio support but the paper does not evaluate it.
  • Evaluation is limited to multiple-choice and constrained open-ended formats; free-form long-form generation quality is not assessed.

Relevance to Vision-Language Models

This work directly advances the frontier of VLM reasoning by demonstrating that GRPO-style RL—previously confined to images or short clips—can be scaled to genuinely long videos via infrastructure innovation (MR-SP) rather than model architecture changes, making the gains broadly applicable to any VLM backbone. The LongVideo-Reason dataset and the CoT-SFT → RL pipeline establish a reproducible recipe for instilling structured temporal, spatial, and causal reasoning into VLMs, filling a significant gap left by math/code-focused RL datasets. The finding that reasoning capability scales with input frame count (monotonically for RL-trained models, but plateaus/degrades for SFT-only models) is a key empirical insight for the VLM community. The public release of the training system supporting VILA and Qwen series further lowers the barrier for RL-based VLM research.