Skip to content

VideoSearch-R1: Iterative Video Retrieval and Reasoning via Soft Query Refinement

🕒 Published (v1): 2026-07-01 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VideoSearch-R1 is an agentic framework that couples a video search engine with a VLM in a multi-turn loop, iteratively retrieving candidate videos, verifying query-video alignment, and refining search queries via Soft Query Refinement (SQR)—a continuous latent-space mechanism trained with GRPO. It unifies corpus-level inter-video retrieval with fine-grained intra-video temporal grounding and achieves state-of-the-art on three Video Corpus Moment Retrieval (VCMR) benchmarks.

Problem

Existing video retrieval pipelines treat retrieval as a one-shot preprocessing step decoupled from downstream reasoning: when initial retrieval fails, there is no mechanism to refine the search, causing cascading failures in intra-video tasks (e.g., temporal grounding). Conversely, video agentic frameworks that perform multi-turn reasoning assume the relevant video is already provided, bypassing the retrieval problem entirely. Neither paradigm handles the joint inter-video + intra-video challenge end-to-end.

Method

VideoSearch-R1 operates as a multi-turn agent wrapping an external cross-modal dense retriever (Qwen3-VL-Embedding-2B). At each turn \(t\):

  1. Retrieval: the current query \(q_t\) is passed to the search engine, retrieving \(v_t = \arg\max_{v \in \mathcal{V}} f(q_t)^\top f(v)\).
  2. Verification: the VLM (Qwen3-VL-2B-Instruct) reasons over \(q_t\) and \(v_t\), producing a chain-of-thought trace \(r_t\) and a binary decision \(y_t^\text{ret} \in \{\text{match}, \text{not match}\}\).
  3. Soft Query Refinement (SQR): on mismatch, instead of rewriting the query in text ("hard refinement"), the model autoregressively generates \(N=8\) soft query tokens \(q_t^\text{soft} \in \mathbb{R}^{N \times D}\) in the continuous latent space by projecting the hidden state of each generated token through a linear layer. The refined query is \(q_{t+1} = [q_1 \| q_t^\text{soft}]\).
  4. Temporal grounding: on match, the model predicts start/end timestamps within the retrieved video.

Training is two-stage: - Stage 1 (SFT): CoT traces annotated by Qwen3-VL-30B-A3B-Thinking. Soft query tokens are supervised via InfoNCE contrastive loss \(\mathcal{L}_\text{ret}\), pushing \([q_1 \| q^\text{soft}]\) toward the ground-truth video and away from negatives in the retriever's embedding space. Total: \(\mathcal{L}_\text{SFT} = \mathcal{L}_\text{verif} + \mathcal{L}_\text{ret} + \mathbf{1}_{y^\text{ret}=\text{match}} \mathcal{L}_\text{time}\). - Stage 2 (GRPO): Four reward signals—format compliance (\(R^\text{format}\)), verification accuracy (\(R^\text{verif}\)), retrieval quality (\(R^\text{ret} = \exp(-\mathcal{L}_\text{ret})\)), and temporal IoU (\(R^\text{time}\))—are summed and used to compute per-sample advantages by within-group normalization: \(A_i = (R_i - \text{mean})/\text{std}\).

Maximum inference turns \(T=2\).

Key Contributions

  • VideoSearch-R1: end-to-end agentic framework tightly coupling inter-video retrieval and intra-video temporal grounding through multi-turn VLM–search-engine interaction.
  • Soft Query Refinement (SQR): query refinement in continuous latent space via autoregressive generation of soft tokens, supervised by InfoNCE; produces fewer generated tokens than hard (text-level) refinement while enabling finer-grained adjustments.
  • Joint GRPO optimization: four complementary reward signals that propagate gradients across both retrieval and reasoning stages simultaneously.
  • State-of-the-art results on ActivityNet-FIG, Charades-FIG, and DiDeMo-FIG VCMR benchmarks.

Results

All metrics are VCMR (IoU/R@1) and VR (R@1) against baselines on three datasets:

  • Charades-FIG: VideoSearch-R1 achieves 0.3/R@1=16.5, 0.5/R@1=13.4, 0.7/R@1=8.2; VR R@1=24.6. vs. fine-tuned baseline: 12.9 / 10.4 / 7.2; VR R@1=21.6 (ZS).
  • DiDeMo-FIG: 0.3/R@1=33.3, 0.5/R@1=30.2, 0.7/R@1=19.7; VR R@1=59.0. vs. FT baseline: 23.6 / 22.1 / 16.7; VR R@1=54.8.
  • ActivityNet-FIG: 0.3/R@1=33.8, 0.5/R@1=22.3, 0.7/R@1=12.3; VR R@1=61.1. vs. FT baseline: 29.1 / 19.2 / 11.4; VR R@1=55.1.
  • VideoSearch-R1 outperforms prior dedicated VCMR methods (CONQUER, SQuiDNet) and both ZS and FT VLM baselines across all three datasets and all IoU thresholds.
  • SQR requires substantially fewer generated tokens than hard query refinement (quantitative comparison referenced but exact token counts truncated in provided text).

Limitations

  • Maximum number of retrieval turns is set to \(T=2\); failure to retrieve within this budget results in a zero VCMR score with no fallback.
  • The video search engine is fixed (Qwen3-VL-Embedding-2B) and not jointly fine-tuned; SQR operates in that frozen retriever's embedding space, potentially limiting refinement expressiveness.
  • SFT cold-start uses only 2K samples per dataset, and soft query token supervision relies on the quality of a 30B teacher model for CoT annotations.
  • Evaluation is confined to VCMR; generalization to open-domain video QA or multi-hop retrieval settings is not demonstrated.
  • The paper text provided is truncated; ablation results on token efficiency and per-turn retrieval gain are referenced but not fully visible.

Relevance to Agentic AI / LLM Agents

VideoSearch-R1 is a concrete instance of the tool-augmented, multi-turn agentic paradigm applied to multimodal retrieval: the VLM acts as a reasoning-and-decision core that dynamically invokes an external search engine, evaluates tool outputs, and issues refined tool calls—mirroring patterns pioneered by Search-R1 and RAG-based agents in NLP. The key novel contribution for the agentic community is SQR—soft query refinement in latent space—which generalizes the idea of "soft reasoning" (e.g., Coconut) from internal chain-of-thought compression to external tool-call parameterization, showing that agent actions need not be discrete text strings. The joint GRPO training over format, verification, retrieval, and grounding rewards demonstrates a multi-objective RL recipe for training agents whose reward spans both tool invocation quality and downstream task success, a design pattern directly transferable to other retrieval-augmented agents.