FrameThinker: Learning to Think with Long Videos via Multi-Turn Frame Spotlighting

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; multi-turn frame spotlighting for efficient long-video VLM reasoning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FrameThinker introduces multi-turn, active frame selection for long video reasoning in LVLMs, replacing static uniform sampling with an iterative "think → select frames → observe" loop. A two-phase SFT+GRPO training pipeline teaches the model when and where to zoom in on relevant video segments. The resulting 7B model achieves state-of-the-art accuracy on multiple long-video benchmarks while processing over 20× fewer frames than comparable baselines.

Problem

Existing LVLMs process a fixed, uniformly sampled set of frames before any reasoning begins, making them inefficient for long videos and susceptible to noise from irrelevant frames. Video agents that can interact with video content rely on predefined external workflows and are not end-to-end learnable. The field lacked a principled, trainable framework for active, iterative visual information gathering during reasoning.

Method

FrameThinker operates as a multi-turn reasoning loop. At each step the model generates a structured output <think>...</think><action>...</action> from a three-action space: (1) choose frames between START_FRAME and END_FRAME — retrieves 8–12 frames from a temporal segment; (2) get frame number at time MM:SS — converts a human-readable timestamp to a frame index; (3) output answer — terminates the loop. This produces a trajectory \(\tau = ((t_1, a_1, o_1), \ldots, (t_n, a_n, o_n))\).

Training is two-phase: - SFT on 2,392 curated examples teaches action syntax via cross-entropy loss on model-generated tokens only (observations excluded from loss). - RL via GRPO on 28k diverse examples optimizes the policy. The reward is:

\[R_\text{final} = R_\text{total} \cdot \mathcal{V}_\text{CCV}(\tau), \quad R_\text{total} = R_\text{acc} + R_\text{action}\]

\[R_\text{action} = \lambda_{cf} \cdot \mathbf{I}(cf \in \tau) + \lambda_{gfn} \cdot \mathbf{I}(gfn \in \tau)\]

where action bonuses are conditional on correct final answers (\(R_\text{acc}=1\)), \(\lambda_{gfn} \gg \lambda_{cf}\) (0.5 vs 0.02 at large scale), and no format reward is used (it suppresses action exploration). The Cognitive Consistency Verification (CCV) module is a rule-based filter applied post-rollout that checks for redundancy, logical flow, and fidelity between thought content and action (e.g., verifying the frame interval in the action matches the interval reasoned about in the thought). Trajectories failing CCV receive zero reward; at inference, CCV triggers a retry/fallback.

Key Contributions

• Multi-turn, end-to-end trainable video reasoning framework (FrameThinker) that constructs a multimodal chain-of-thought interleaving text and retrieved frames.
• Comprehensive empirical exploration of RL reward design for multi-turn video: demonstrates that (i) format rewards suppress action usage, (ii) unconditional action bonuses cause mode collapse, (iii) conditional bonuses alone still reinforce illogical thought-action pairs, motivating CCV.
• Cognitive Consistency Verification (CCV) module that enforces logical coherence between reasoning traces and actions during both training and inference.
• State-of-the-art results on six long-video benchmarks with dramatic frame-efficiency gains.

Results

• LongVideo-Reason: 76.1% accuracy at avg. 20.6 frames vs. LongVILA-R1 72.0% at 512 frames (>20× fewer frames).
• Video-Holmes: 46.8% overall accuracy (new SOTA); 56.1% on small-scale training subset, vs. Video-R1 36.5% and GRPO-CARE* 40.7%, using avg. 10.2 frames vs. 32 frames for baselines.
• Average across all 6 benchmarks: +10.4% over baselines.
• Ablations confirm CCV improves performance beyond interpretability, conditional reward outperforms unconditional, and \(\lambda_{gfn} = 0.2, \lambda_{cf} = 0\) is optimal at small scale.

Limitations

• Action space is small and manually defined (3 actions); more complex tool use is not explored.
• Frame selection granularity is fixed (8 or 12 frames per turn), which may still miss fine-grained temporal events.
• The CCV module uses hand-crafted consistency rules; these may not generalize to all reasoning patterns or video domains.
• Evaluated only on a 7B parameter model (Qwen2.5-VL-7B-Instruct); scalability to larger models is not demonstrated.
• RL training requires 28k examples and multi-turn rollout, which is more resource-intensive than standard RLVR pipelines.

Relevance to Vision-Language Models

FrameThinker directly advances VLM capability for long-form video understanding by shifting the paradigm from passive, one-shot visual encoding to active, iterative visual querying — a "thinking with videos" analog to recent "thinking with images" work for static VLMs. The GRPO-based training pipeline and reward design findings (particularly the failure modes of unconditional/format rewards and the necessity of CCV for logical consistency) provide actionable guidance for anyone applying RLVR to multimodal, multi-step tasks. The efficiency result — matching or surpassing models using 20× more frames — has direct implications for long-context VLM deployment. This work extends the emerging thread of agentic, self-directed visual reasoning that is reshaping how the community thinks about VLM inference-time compute.