Skip to content

Bridging VideoQA and Video-Guided Agentic Tasks via Generalized Keyframe Extraction

🕒 Published (v1): 2026-06-28 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper identifies that existing video benchmarks only test shallow perception, not whether MLLMs can learn procedural skills from tutorials and generalize to agentic tasks. It introduces VG-GUI-Bench (1,000 test cases pairing video tutorials with GUI agent tasks) and proposes TASKER, a training-free keyframe extraction algorithm that treats frame selection as a graph-search problem with task-driven and scene-aware cost functions. TASKER achieves SOTA on VideoQA (EgoSchema +2.0%, NExT-QA +1.8%) and uses ~1/4 the frames of VideoTree at equivalent accuracy.

Problem

  • Existing VideoQA benchmarks only evaluate shallow perceptual cues (objects, attributes, short-term actions), not whether MLLMs can extract procedural knowledge from video tutorials and transfer it to long-horizon agentic tasks (video in-context learning).
  • Model performance on both VideoQA and video-guided agentic tasks critically depends on how keyframes are selected — naive uniform sampling misses critical moments, while exhaustive processing is computationally wasteful. Using GPT-4o with a good frame selection strategy improves accuracy by ~15% over uniform sampling with the same frame budget.
  • No high-quality benchmark exists for systematically evaluating MLLM-based GUI agents on video-guided, long-horizon GUI task execution.

Method

VG-GUI-Bench: 1,000 test cases built from MONDAY dataset, each pairing a tutorial video with a GUI task requiring similar procedural knowledge. Defines a standardized 6-action space: CLICK(x,y), SCROLL(x1,y1,x2,y2), TYPE(content), PRESS(key∈{BACK,HOME,ENTER}), ZOOM(), FINISH(). Four metrics: Accuracy (type + argument correctness), Completion (proportion of correct steps per episode), Efficiency (frames per prediction step), and PIR (relative gain from video input).

TASKER (Task-driven And Scene-aware Keyframe searchER): A training-free, zero-shot keyframe extraction algorithm that treats video frame selection as a graph-search problem over a tree of video segments: - Divides the video into \(M\) uniform segments (initial nodes), then iteratively selects and binary-splits the most promising segment. - Cost functions inspired by classical search: TASKER-GBFS (task-driven — MLLM estimates which segment contains missing information), TASKER-Dijkstra (scene-aware — prioritizes segments with largest scene/activity change), TASKER-A (sum of both), TASKER-BFS (naive breadth-first, no MLLM cost evaluation). - Termination* via dual-confidence voting: the MLLM self-evaluates its answer (confidence \(c_1\)) and also predicts from a temporally-summarized caption (confidence \(c_2\)); search stops only when both exceed threshold \(C\). - A frame validation step discards redundant or task-irrelevant new frames, adding barren segments to a frozen set \(S_{\text{frozen}}\) to avoid re-exploration.

Key Contributions

  • Proposes a two-level taxonomy (VideoQA → Video-Guided Agentic Tasks) that reframes procedural video understanding as a form of video in-context learning.
  • Introduces VG-GUI-Bench, a benchmark with standardized action space and four complementary metrics for evaluating video-guided GUI agent tasks.
  • Proposes TASKER, a generalized, training-free keyframe extraction algorithm unifying task-driven relevance and scene-aware dynamics via graph-search-inspired cost functions.
  • Demonstrates SOTA results on EgoSchema (63.1%) and NExT-QA (77.4%) with high frame efficiency (~15% of total frames processed; at 66% accuracy, uses ~1/4 the frames of VideoTree).

Results

  • EgoSchema fullset: TASKER (GPT-4) achieves 63.1% accuracy, +2.0% over best baseline (VideoTree 61.1%). With GPT-4o: 63.6%, +2.5%.
  • NExT-QA average: TASKER (GPT-4) achieves 77.4%, +1.8% over best baseline. With GPT-4o: 78.1%, +2.5%.
  • Frame efficiency: At the same 66% accuracy on EgoSchema subset, TASKER uses ~1/4 the frames of VideoTree. TASKER processes only ~15% of total frames vs. methods like LangRepo/LifelongMemory that process all frames.
  • Open-source MLLM: TASKER (Qwen3-VL-235B-A22B) also consistently outperforms VideoTree and VideoAgent across all NExT-QA sub-questions (Temporal, Causal, Descriptive).
  • VG-GUI-Bench leaderboard (7 frontier models): Gemini-3.1-Pro ranks #1 (Acc. 61.68% with 10 uniform frames, PIR 0.054); GPT-5-mini ranks #2 (58.40%, PIR 0.047). Adding 10 uniform frames consistently improves over no-video across all models.
  • TASKER is training-free and zero-shot, yet outperforms training-based methods (LVNet, Vamos).

Limitations

  • TASKER only accesses visible frames — it does not utilize information from unselected frames, which could miss subtle cues.
  • Cost function evaluation relies on MLLM calls at each expansion step, introducing computational overhead proportional to search depth.
  • Binary-split segment expansion is a coarse partitioning strategy; content boundaries may not align with natural shot/scene transitions.
  • The confidence threshold \(C\) is a tunable hyperparameter with no principled default — too low causes early termination, too high wastes frames.
  • VG-GUI-Bench is limited to mobile GUI tasks (1,000 cases); generalization to other domains (web, desktop, robotics) is not evaluated.

Relevance to Agentic AI / LLM Agents

This work directly targets the gap between passive video understanding and active agentic execution — a core challenge for LLM-based agents that must learn from demonstrations. TASKER provides a plug-and-play frame selection module that improves downstream agentic task performance without fine-tuning, while VG-GUI-Bench offers a standardized evaluation protocol for video-guided GUI agents. The framing of "video in-context learning" connects procedural video understanding to the broader in-context learning paradigm in LLM agents, and the finding that frame selection quality dominates downstream accuracy has implications for any agentic system consuming long video inputs.