Skip to content

LensWalk: Agentic Video Understanding by Planning How You See in Videos

🕒 Published (v1): 2026-03-25 17:38 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; agentic lens planning for dense temporal video understanding

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LensWalk is a training-free agentic framework that gives an LLM reasoner explicit control over temporal scope and sampling density when observing video, establishing a reason–plan–observe loop rather than a static one-shot perception stage. At each step the reasoner issues a parameterized tool call specifying which interval to inspect and at what granularity, and a VLM-based observer executes that call on raw video frames. Without any fine-tuning, LensWalk improves o3 accuracy by 11.5% on LVBench and 6.7% on VideoMME-Long.

Problem

Existing video understanding pipelines decouple reasoning from perception: frames are sampled or compressed once before inference, so the visible context cannot adapt as intermediate hypotheses evolve. Retrieval-based agentic methods expand this fixed context via pre-processed artifacts (captions, clip embeddings), but still cannot control temporal granularity or revisit ambiguous raw footage on demand. This mismatch causes missed sparse decisive events, wasted budget on irrelevant segments, and inability to resolve complex multi-step questions.

Method

LensWalk organizes inference as an iterative loop where an LLM Reasoner \(M_r\) issues structured observation actions and a VLM Observer \(M_o\) executes them on raw video:

\[a_t = M_r(Q,\,V_{\text{info}},\,L_{t-1}), \quad a_t := (o_t, q_t, I_t, \rho_{o_t})\]

where \(o_t\) selects a tool, \(q_t\) is a guiding sub-question, \(I_t\) specifies temporal interval(s) and frame budget, and \(\rho_{o_t}\) holds tool-specific arguments. The observation \(e_t\) is appended to history \(L_t\).

The Observation Toolkit has three tools: - Scan Search: Partitions a broad interval into slices, sparsely samples each (180 frames/call), and queries \(M_o\) per slice in parallel—suited for initial localization. - Segment Focus: Densely samples a single continuous interval (32 frames/call) for fine-grained attribute extraction or hypothesis verification; can be re-applied with higher density to the same segment. - Stitched Verify: Concatenates frames from multiple non-contiguous segments \(S = [(s_i, e_i, r_i), \ldots]\) with per-segment sampling rates (128 frames/call total) into one coherent VLM context, enabling asymmetric sampling and cross-segment causal verification in a single pass.

Two lightweight memory mechanisms maintain cross-turn coherence: Timestamp Anchors embed explicit time references (e.g., "01:15–01:40") as text within the frame sequence to ground the observer's outputs; a Subject Memory Table is updated by the LLM after each turn, storing entity descriptions and seen timestamps as a structured knowledge base that is prepended to each reasoning step, avoiding repeated re-identification from long histories.

Key Contributions

  • A reason–plan–observe architecture where temporal scope and sampling density are first-class action parameters determined by the evolving chain of thought, not fixed preprocessing.
  • Three complementary observation tools (Scan Search, Segment Focus, Stitched Verify) enabling broad localization, dense inspection, and non-contiguous multi-segment verification.
  • Timestamp Anchors and a Subject Memory Table as lightweight grounding mechanisms that maintain entity consistency across turns without heavy pre-scanning.
  • Plug-and-play design: no model fine-tuning; works with any Reasoner/Observer combination.
  • Emergent budget self-regulation: easy queries resolve in few turns; genuinely uncertain cases receive more tool calls, with total token cost far below retrieval agents.

Results

All numbers are accuracy (%) from Tables 1–3 unless noted.

  • LVBench: LensWalk (o3) 68.6 vs. o3 standalone 57.1 (+11.5 pp); vs. MR.Video 60.8; vs. Deep Video Discovery 74.2.
  • VideoMME Long (w/o sub): LensWalk (o3) 71.4 vs. o3 standalone 64.7 (+6.7 pp); vs. MR.Video 61.8; vs. Deep Video Discovery 67.3 (+4.1 pp).
  • LongVideoBench (Val Long): LensWalk (o3) 70.6 vs. o3 60.6.
  • EgoSchema: LensWalk (o3) 74.8 vs. o3 63.2.
  • MMVU (MC): LensWalk (o3) 79.2 vs. o3 78.9; LensWalk (o3/GPT-4.1) 80.9.
  • Video-MMMU Overall: LensWalk (o3) 78.33 vs. o3 75.44; LensWalk (o3/GPT-4.1) 77.11.
  • VideoMME Long, observer ablation: GPT-4.1 observer with o3 reasoner 70.0 (+6.9 over GPT-4.1 alone 63.1); Qwen2.5-VL-7B observer with o3 reasoner 61.3 (+5.9 over 55.4 baseline).
  • GPT-5 in LensWalk: LVBench 66.9, VideoMME Long 69.2.

Limitations

  • Reasoner quality is the critical bottleneck: open-source Qwen3-235B-A22B as reasoner yields only +0.1% on GPT-4.1 observer and −0.6% on Qwen2.5-VL-72B observer (VideoMME Long), unable to improve strong observers.
  • Observer and memory-updater errors can introduce entity inconsistencies across turns that the lightweight Subject Memory Table cannot always resolve.
  • Fixed per-call frame caps (180/32/128) and a hard ceiling of 20 tool invocations per query constrain very long or dense videos.
  • Relies on proprietary APIs (o3, GPT-4.1, GPT-5) for its strongest results; open-source combinations show smaller and less consistent gains.

Relevance to Vision-Language Models

LensWalk reframes the VLM's role in video understanding: instead of being queried once over a pre-fixed context, the VLM becomes a parameterized observer whose inputs—temporal interval, sampling density, segment composition—are decided step-by-step by a language reasoner. This demonstrates that VLMs' visual grounding ability is more effectively exploited through fine-grained, reasoning-conditioned observation scheduling than through context window maximization. The plug-and-play design and consistent cross-model gains (Qwen2.5-VL-7B through o3) make LensWalk a broadly applicable test-time scaling strategy for any VLM deployed on long-video tasks. It also highlights a new design axis—active perception planning—distinct from prior VLM work on retrieval augmentation or token compression.