Skip to content

Deep Video Discovery: Agentic Search with Tool Use for Long-form Video Understanding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Deep Video Discovery (DVD) is an agentic system for long-form video understanding that equips an LLM reasoning core with three hierarchical search tools—Global Browse, Clip Search, and Frame Inspect—operating over a pre-built multi-granular video database. Unlike prior video agents with fixed search workflows, DVD autonomously plans adaptive tool-use chains per query. It achieves 74.2% accuracy on LVBench, surpassing all prior work by 13.4%.

Problem

Hour-long videos exceed the effective context capacity of VLMs even at million-token context lengths, and existing video agents address this with manually prescribed, rigid search workflows (e.g., tree traversal in VideoTree/VCA) that cannot adapt to diverse query types—fine-grained local queries and holistic global queries require fundamentally different retrieval strategies.

Method

DVD operates in two stages. First, an offline multi-granular video database is built by segmenting the video into 5-second clips, captioning each with a VLM while maintaining a progressive subject registry (name, appearance, temporal span), embedding captions with a dense encoder, and retaining raw decoded frames at 2 fps. The database encodes three levels: global subject/event summaries, per-clip text captions with embeddings, and indexed frames.

Second, an Agentic Search and Answer (ASA) loop runs at query time. An LLM (OpenAI o3) serves as orchestrator in a ReAct-style observe-reason-act loop with three tools: - Global Browse: retrieves pre-built subject-centric summary + on-demand event-centric summary (frames sampled globally, VLM prompted with user query). - Clip Search: semantic retrieval via cosine similarity between agent-synthesized query embedding and clip caption embeddings; returns top-k captions and timestamps. - Frame Inspect: loads raw frames from a specified temporal interval and runs open-format VQA via a VLM (also o3), capped at 50 frames.

The agent iterates up to N=15 steps, accumulating history, until it emits an ANSWER action or hits the step limit. No fixed workflow is prescribed; tool sequencing emerges from LLM reasoning.

Key Contributions

  • Adaptive, query-conditioned agentic search over long video without hardcoded workflow—tool chains emerge from LLM planning.
  • Three-level hierarchical video database (global/clip/frame) enabling tools at matching granularities.
  • State-of-the-art on LVBench (74.2% / 76.0% with transcripts), outperforming the prior SOTA MR. Video by 13.4% and the base VLM (o3 alone) by 17.1%.
  • Behavioral analysis categorizing agent trajectories into five types (Global Browse Only, Simple Action, Iterative Search, Frame Inspect Trap, Clip Search Trap), yielding design insights on reasoning depth vs. overconfidence.

Results

  • LVBench: DVD 74.2% (76.0% with transcripts) vs. prior SOTA MR. Video 60.8%, VCA 41.3%, OpenAI o3 (256-frame baseline) 57.1%.
  • LongVideoBench (Long subset, >900s): DVD 68.6% vs. prior SOTA 61.6% (+7.0%).
  • Video MME Long (no subtitles): DVD 67.3% vs. AdaRETAKE 65.0% (+2.3%), vs. MR. Video 61.8% (+5.5%).
  • EgoSchema val: DVD 76.6%, exceeding reported human-level accuracy (~76%) and prior best by 3.0%.
  • Ablation: removing Clip Search causes the largest drop (−12.3%); Frame Inspect −8.4%; Global Browse −2.9%.
  • Adaptive workflow outperforms VideoAgent's fixed workflow by 7.5% at comparable step counts; fixed workflow at 11.1 avg. steps (vs. DVD's 7.3) still trails by 4.0%.
  • Open-source reasoning: DVD + DeepSeek-R1 achieves 68.5%, surpassing all prior methods; DVD + Qwen3-32B achieves 57.3%, outperforming GPT-4o and o3 baselines.

Limitations

  • Iterative agentic search incurs substantially higher computational cost than single-pass VLM inference.
  • Clip Search Trap and Frame Inspect Trap are identified failure modes where the agent loops unproductively, particularly when key information is absent from the database.
  • Pre-built captions introduce irreversible information loss; fine-grained visual details not captured in captions require Frame Inspect, increasing latency.
  • Performance depends critically on the reasoning capability of the backbone LLM—weaker models (GPT-4o) collapse to shallow "Simple Action" patterns with 91.4% frequency, causing a 13.7% accuracy drop.
  • API content filtering (Azure OpenAI safety layer) reduces measurable performance on some benchmark items.

Relevance to Agentic AI / LLM Agents

DVD is a direct application of the tool-augmented agentic paradigm (ReAct, Deep Research) to a structured multimodal retrieval problem, demonstrating that replacing fixed workflows with emergent LLM planning yields substantial gains even when the environment (video) is complex and high-dimensional. The 17.1% gain over the same underlying LLM used as a standard VLM quantifies the contribution of the agentic loop itself, independent of model capability. The behavioral taxonomy (Simple Action vs. Iterative Search vs. Trap states) provides concrete empirical evidence about how reasoning depth, overconfidence, and tool-loop failure modes interact in real agentic deployments—directly relevant to designing robust tool-use agents. The finding that stronger reasoning models (o3 > o4-mini > GPT-4o) drive dramatically different trajectory patterns aligns with broader arguments that agentic system quality is bottlenecked by the orchestrator's planning ability rather than individual tool quality.