Skip to content

LensWalk: Agentic Video Understanding by Planning How You See in Videos

🕒 Published (v1): 2026-03-25 17:38 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LensWalk is a training-free agentic framework that replaces static, one-shot video frame sampling with a multi-round reason–plan–observe loop, letting an LLM reasoner actively control which temporal segments to inspect and at what sampling density at each reasoning step. A compact three-tool observation suite (Scan Search, Segment Focus, Stitched Verify) drives fresh VLM queries against raw video rather than preprocessed surrogates. The framework is plug-and-play, boosting o3 by 11.5% on LVBench and 6.7% on VideoMME-Long with no fine-tuning.

Problem

Existing video understanding pipelines divorce reasoning from perception: they convert a video into a fixed context (uniformly sampled frames, captions, or clip embeddings) before reasoning begins, so the observation budget cannot be reallocated as intermediate hypotheses change. Retrieval-based agents query preprocessed artifacts (ASR, OCR, dense captions) rather than the raw video, providing no control over temporal granularity or segment composition during inference.

Method

LensWalk instantiates video understanding as an iterative agent loop with three components:

Reasoner (\(M_r\)): At step \(t\), the LLM reads query \(Q\), video metadata \(V_\text{info}\), and history \(L_{t-1}\) to emit a parameterized action \(a_t = (o_t, q_t, I_t, \rho_{o_t})\) specifying the tool, a guiding sub-question, temporal scope/frame budget, and tool-specific arguments.

Observation Toolkit (\(\mathcal{O}\)): Three VLM-powered tools execute the plan on raw video: - Scan Search: partitions a broad interval into \(n_s\) slices, sparsely samples each, and queries the VLM per-slice in parallel — for coarse cue discovery. - Segment Focus: densely samples a single continuous interval for fine-grained attribute extraction or hypothesis verification. - Stitched Verify: assembles frames from non-contiguous segments \(S = [(s_i, e_i, r_i), \ldots]\) with per-segment sampling rates into one batch — for cross-segment causal or comparative checking.

Evidence Grounding: Two lightweight memory mechanisms maintain coherence across turns: (1) Timestamp Anchors inserted into frame sequences so the VLM observer grounds responses with explicit temporal references; (2) a Subject Memory Table maintained outside the main history that tracks entities (descriptions, intervals) using canonical identifiers, preventing repeated re-identification and redundant context tokens.

The loop runs for up to 20 tool calls; each action-observation pair is appended to \(L_t\), transforming the history into a temporally tagged structured document that guides under-explored region targeting in subsequent steps.

Key Contributions

  • A reason-scheduled active observation paradigm that treats context selection as a step-wise, tool-mediated process driven by evolving hypotheses, rather than a one-time preprocessing step.
  • A minimal yet expressive three-tool observation suite (Scan Search, Segment Focus, Stitched Verify) covering broad discovery, fine-grained inspection, and cross-segment integration with asymmetric sampling rates.
  • Timestamp Anchors + Subject Memory Table as lightweight, structured evidence grounding that avoids heavy pre-scanning or intricate bookkeeping used in prior agentic systems.
  • A plug-and-play, training-free architecture compatible with any Reasoner/Observer VLM pair; validated across proprietary (o3, GPT-4.1, GPT-5) and open-source (Qwen2.5-VL-7B/72B, Qwen3-235B-A22B) models.
  • Emergent adaptive efficiency: the agent naturally resolves easy queries in fewer turns and allocates more budget to ambiguous cases, achieving token usage far below retrieval agents.

Results

  • LensWalk (o3/o3): 68.6% on LVBench (vs. o3 standalone 57.1%, +11.5%), 71.4% VideoMME-Long (vs. o3 64.7%, +6.7%).
  • LensWalk (GPT-5/GPT-5): 66.9% LVBench, 69.2% VideoMME-Long (vs. GPT-5 baseline 59.8% and 68.4%).
  • Outperforms all prior agentic methods including Deep Video Discovery (74.2% vs. LensWalk 66.8% on LVBench with o3/GPT-4.1) except Deep Video Discovery on LVBench; surpasses Deep Video Discovery on VideoMME-Long (71.4% vs. 67.3%) by >4%.
  • On reasoning benchmarks: LensWalk (o3) achieves 79.2% MMVU MC and 78.33% Video-MMMU Overall (vs. o3 standalone 78.9% and 75.44%).
  • Open-source Reasoner Qwen3-235B-A22B improves Qwen2.5-VL-7B Observer by +4.3% on VideoMME-Long but fails to improve strong observers (GPT-4.1: +0.1%, Qwen2.5-VL-72B: -0.6%).
  • Evaluated on: LVBench, LongVideoBench (long val), VideoMME (30–60 min long split), MMVU, Video-MMMU, EgoSchema.

Limitations

  • Capped at 20 tool invocations per query; genuinely complex videos requiring many observation rounds may hit this ceiling.
  • Reasoner quality is the decisive bottleneck: weaker open-source reasoners (Qwen3-235B-A22B) fail to improve strong observers, suggesting the framework's gains are tightly coupled to Reasoner capability.
  • Observer/memory-updating errors can introduce inconsistencies in the Subject Memory Table; the paper acknowledges this but relies on re-observation for self-correction rather than a stronger consistency guarantee.
  • Experiments cover only multiple-choice QA benchmarks; open-ended generation or temporal grounding tasks are not assessed.
  • Per-call frame budgets (180 / 32 / 128 for Scan/Focus/Stitch) are fixed hyperparameters, not dynamically allocated by the agent.

Relevance to Harnesses / Meta-Harnesses

LensWalk is a concrete instance of a tool-orchestrating meta-harness applied to the video-understanding domain: the LLM acts as a meta-controller that selects, parameterizes, and sequences VLM-powered observation tools at runtime rather than executing a fixed pipeline. The reason–plan–observe loop is structurally analogous to agentic harnesses that wrap specialized sub-tools (retrievers, analyzers) under a single reasoning agent, making LensWalk a transferable architectural template. Its plug-and-play, model-agnostic design — where any Reasoner/Observer pair can be swapped in — mirrors how meta-harnesses abstract over backend executors. The lightweight grounding mechanisms (Subject Memory Table, Timestamp Anchors) address the cross-turn state management problem central to multi-step harness design, offering reusable patterns for coherent evidence accumulation without heavy preprocessing infrastructure.