Skip to content

Temporal Chain of Thought: Long-Video Understanding by Thinking in Frames

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Temporal Chain of Thought (TCoT) is a training-free inference strategy for video question-answering that uses a single VLM to iteratively select the most relevant frames from a video before answering, rather than feeding all frames at once. By curating the input context, TCoT outperforms long-context VLM baselines while using smaller context windows, achieving state-of-the-art on four video QA benchmarks.

Problem

Long-context VLMs (supporting ~1000 frames) degrade in accuracy when the input context contains irrelevant or distracting frames — a "lost-in-the-middle" effect well-documented for LLMs. Existing multi-model pipelines that address this (e.g., captioning + retrieval + LLM answering) introduce information loss through caption intermediaries and require multiple heterogeneous models. Processing very long videos (>1 hour) is infeasible within any fixed context window under standard uniform sampling.

Method

TCoT decomposes video QA into two stages using a single instruction-tuned VLM: 1. Context aggregation G(x, q): The VLM is prompted (zero-shot, JSON output) to return frame IDs relevant to the question, along with a textual justification. 2. Answering H(c, q): The VLM answers using only the selected frames, augmented with a small number u of uniformly sampled frames to provide coarse global context.

For long videos, Dynamic-Segment TCoT partitions the video into l non-overlapping segments, runs Single-Step TCoT (selecting s frames) on each segment independently, concatenates the per-segment selections, then refines if the union exceeds the context limit k. This decouples required context per VLM call (always s frames ≤ 32K tokens) from total video length, and the number of segments l is a tunable compute–accuracy knob. A Hierarchical variant (coarse-to-fine iterative selection) is also evaluated but underperforms the partitioned approach on long videos.

Key Contributions

  • Training-free, single-model inference strategy: no auxiliary captioners, retrieval encoders, or multi-model pipelines required.
  • Dynamic-Segment TCoT that processes arbitrarily long videos at fixed per-call context cost by partitioning into l segments, enabling inference-time scaling by varying l.
  • Demonstration that VLM-based direct frame selection outperforms feature-similarity retrieval (SigLIP) and caption-based selection.
  • Interpretable intermediate output: the VLM's justification for frame selection can be inspected.
  • State-of-the-art results on EgoSchema, LVBench, NExT-QA, and OpenEQA with three different VLMs (Gemini 1.5 Flash, Qwen-2.5-VL-7B, GPT-4o-mini).

Results

  • LVBench (avg. 68 min videos): TCoT (Gemini 1.5 Flash, 32K context) scores 61.7 vs. baseline 50.3 (+11.4 pts); at matched 700K total token budget, TCoT 61.7 vs. baseline 58.9 (+2.8 pts). TCoT (GPT-4o-mini) improves by +5.5 pts (53.5 vs. 48.0).
  • EgoSchema: TCoT achieves 75.2 (subset) / 69.1 (full set) vs. prior best 68.2 / 67.8 (LongVU).
  • NExT-QA: TCoT 81.0 vs. prior best 80.0 (Gemini 1.5 Flash baseline).
  • OpenEQA: TCoT 69.2 vs. Gemini 1.5 Flash baseline 68.0 and GPT-4V 55.3.
  • Context aggregation ablation (LVBench): Dynamic-Segment 61.7 > Hierarchical 53.3 > Baseline 50.3 > Single-Step 48.3.
  • Compute–accuracy: TCoT accuracy scales smoothly with l (number of segments); baseline inference saturates around 1000 frames (264K tokens) and self-consistency CoT provides negligible gains.
  • Frame selection analysis: VLM direct-from-frames selection (61.7) outperforms VLM from "long" captions (60.4), from "concise" captions (58.3), and SigLIP feature similarity (54.4); oracle human-annotated time references yield 67.4.

Limitations

  • Requires a VLM with strong instruction-following and zero-shot JSON output capability to perform reliable frame selection; weaker models may fail the selection call (fallback: use all frames).
  • No explicit training for the selection task; future work could fine-tune via RL.
  • Dynamic-Segment TCoT may miss very short events not captured in the s-frame uniform sample from a segment (shown by failure cases in Fig. 5).
  • At very low total frame budgets on LVBench (<512 frames), TCoT underperforms long-context baseline because too few frames are available to identify relevant ones.
  • Evaluated only on multiple-choice and open-ended QA; generalization to other video understanding tasks (e.g., dense captioning, temporal localization as output) is not shown.

Relevance to Vision-Language Models

TCoT directly addresses a fundamental limitation of long-context VLMs — their inability to attend selectively over thousands of frames — by repurposing the VLM's own instruction-following capability as an adaptive retriever, without any architectural change or fine-tuning. This positions TCoT as a plug-in inference-time module compatible with any sufficiently capable VLM, demonstrating that inference-time compute scaling (via increasing l) is a more effective strategy for long-video understanding than simply expanding the context window. The work reinforces the connection between Chain-of-Thought reasoning in LLMs and structured intermediate representations in video — here, selected frame indices serve as "visual thoughts." For researchers tracking VLMs, TCoT illustrates how the context-distractor problem is a first-order concern at scale, and that self-referential prompting strategies can close much of the gap before RL-based training is applied.