Skip to content

VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinement

🕒 Published (v1): 2026-07-01 04:59 UTC · Source: Arxiv · Venue: ECCV 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VideoSearch-R1 is an agentic framework that couples a Vision-Language Model with an external video search engine in a closed multi-turn loop, unifying corpus-level video retrieval (inter-video reasoning) with fine-grained temporal grounding within retrieved videos (intra-video reasoning). The key novelty is Soft Query Refinement (SQR), which refines retrieval queries in a continuous latent space rather than rewriting them as text. The model is trained end-to-end via SFT followed by GRPO with composite task-level rewards, achieving state-of-the-art on three Video Corpus Moment Retrieval benchmarks.

Problem

Existing video understanding pipelines decouple retrieval from reasoning: retrieval runs once as a preprocessing step, and downstream intra-video modules receive its output unconditionally. When the initial retrieval fails, the error propagates with no recovery path. Simultaneously, existing video agentic frameworks assume the relevant video is already known, bypassing the retrieval stage entirely. No prior work jointly optimizes iterative retrieval and intra-video reasoning in an end-to-end trainable loop.

Method

Architecture. VideoSearch-R1 wraps Qwen3-VL-2B-Instruct around an external dense video retriever (Qwen3-VL-Embedding-2B). At each turn \(t\), the model receives the current query \(q_t\) and top-1 retrieved video \(v_t = \arg\max_{v \in \mathcal{V}} f(q_t)^\top f(v)\), produces a chain-of-thought reasoning trace, outputs a binary match/mismatch verdict, and either proceeds to temporal grounding (match) or triggers SQR (mismatch).

Soft Query Refinement (SQR). Instead of generating new text (hard refinement), the model autoregressively produces \(N=8\) soft query tokens \(q_t^{\text{soft}} \in \mathbb{R}^{N \times D}\) by projecting hidden states through a linear layer and feeding them directly as input embeddings. The refined query is \(q_{t+1} = [q_1 \| q_t^{\text{soft}}]\), appended to the original query embedding and passed back to the retriever.

Training — Stage 1 (SFT cold start). 2K query-video pairs (1K positive, 1K negative) are used. Positive pairs train \(\mathcal{L}_{\text{verif}}\) and \(\mathcal{L}_{\text{time}}\); negative pairs train \(\mathcal{L}_{\text{verif}}\) plus an InfoNCE contrastive loss on soft query tokens: $\(\mathcal{L}_{\text{ret}} = -\log \frac{\exp(f([q_1 \| q^{\text{soft}}])^\top f(v))}{\exp(f([q_1 \| q^{\text{soft}}])^\top f(v)) + \sum_{v^- \in \mathcal{V}^{\text{neg}}} \exp(f([q_1 \| q^{\text{soft}}])^\top f(v^-))}\)$ Overall: \(\mathcal{L}_{\text{SFT}} = \mathcal{L}_{\text{verif}} + \mathcal{L}_{\text{ret}} + \mathbb{1}[y^{\text{ret}} = \text{`match'}] \cdot \mathcal{L}_{\text{time}}\).

Training — Stage 2 (GRPO). RL with four rewards: \(R^{\text{format}}\) (structural compliance), \(R^{\text{verif}}\) (binary match accuracy), \(R^{\text{ret}} = \exp(-\mathcal{L}_{\text{ret}})\) (retrieval quality via reused InfoNCE), and \(R^{\text{time}}\) (temporal IoU). Total reward \(R_i = R_i^{\text{format}} + R_i^{\text{verif}} + R_i^{\text{ret}} + \mathbb{1}[y^{\text{ret}}=\text{`match'}] \cdot R_i^{\text{time}}\); advantages normalized within rollout groups (\(G=8\)). Maximum turns \(T=2\) at inference.

Key Contributions

  • VideoSearch-R1: first agentic framework that jointly trains iterative corpus-level video retrieval and intra-video temporal grounding in a single model via multi-turn tool interaction.
  • Soft Query Refinement (SQR): continuous-latent query refinement trained with InfoNCE, achieving finer-grained adjustments with fewer generated tokens than text-level (hard) refinement.
  • Multi-reward GRPO training: composite reward signal spanning format, verification, retrieval (soft), and temporal grounding, enabling end-to-end optimization across the full retrieval-reasoning loop.
  • SOTA on VCMR: first system to advance both the retrieval (VR) and end-to-end (VCMR) metrics simultaneously across ActivityNet-FIG, Charades-FIG, and DiDeMo-FIG.

Results

All metrics are VCMR (IoU threshold / R@1) or VR (R@1) on three benchmarks vs. same-backbone baselines.

  • Charades-FIG VCMR: VideoSearch-R1 16.5 / 13.4 / 8.2 (IoU 0.3/0.5/0.7 R@1) vs. FT baseline 12.9 / 10.4 / 7.2; vs. SQuiDNet 2.6 / 0.9.
  • DiDeMo-FIG VCMR: 33.3 / 30.2 / 19.7 vs. FT 23.6 / 22.1 / 16.7; vs. SQuiDNet 2.9 / 0.5.
  • ActivityNet-FIG VCMR: 33.8 / 22.3 / 12.3 vs. FT 29.1 / 19.2 / 11.4; vs. SQuiDNet 4.7 / 2.1.
  • VER accuracy: 75.7 (Charades), 74.6 (DiDeMo), 83.3 (ActivityNet) — high verification accuracy drives effective loop termination.
  • SQR token efficiency: SQR requires significantly fewer generated tokens than hard query refinement while achieving better retrieval performance (ablated in paper analyses, exact numbers not fully reproduced in provided text).

Limitations

  • Maximum inference turns fixed at \(T=2\); performance on harder retrieval tasks requiring more iterations is not explored.
  • The SQR soft tokens are not human-interpretable, making failure analysis and debugging of the refinement step opaque.
  • The video search engine (Qwen3-VL-Embedding-2B) is frozen; co-training the retriever could further improve alignment but is not attempted.
  • Evaluation is limited to the VCMR task and three datasets; generalization to open-domain or streaming video corpora is untested.
  • SFT cold-start uses only 2K samples per dataset, which may underfit on more diverse query distributions.

Relevance to Harnesses / Meta-Harnesses

VideoSearch-R1 is a concrete instantiation of a trainable agentic harness: it hard-codes a multi-turn loop structure (retrieve → verify → refine or ground) and trains both the loop's decision logic and the tool-interaction parameters (soft query tokens) end-to-end via RL. This is directly relevant to harness design because it demonstrates how a harness's internal state between tool calls (the refined query embedding) can itself be a learnable, latent object rather than a symbolic plan. The composite GRPO reward spanning format compliance, tool-call quality, and task outcomes is a reusable pattern for training harnesses that must optimize across heterogeneous sub-objectives. For researchers tracking meta-harnesses, the SFT-then-GRPO two-stage bootstrap—using a stronger model (30B) to generate CoT traces for a weaker model (2B) that then self-improves via RL—exemplifies a scalable recipe for bootstrapping harness reasoning without hand-authored trajectories.