Skip to content

Video Action Differencing

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Video Action Differencing (VidDiff) is a new task requiring models to identify fine-grained, natural-language-described differences between two videos of the same action (e.g., expert vs. novice squat). The authors release VidDiffBench (549 video pairs, 4,469 annotated differences across fitness/sports/surgery/music/diving) and propose an agentic three-stage pipeline that outperforms GPT-4o and Gemini-1.5-Pro baselines. Current LMMs perform only modestly above random chance, exposing a hard open problem for video understanding.

Problem

No benchmark or method existed for fine-grained, cross-video action comparison producing natural language feedback. Prior work either used coarse-grained step differences, domain-specific motion visualization, or binary skill-ranking supervision — none produced open-vocabulary, frame-level difference descriptions generalizable across domains and usable for skill coaching.

Method

VidDiff Method is a zero-shot agentic workflow with three sequential stages: 1. Difference Proposer (LLM): Given an action description, an LLM (GPT-4o) generates a set of candidate visually-assessable difference statements. 2. Frame Localizer (LLM + CLIP): An LLM decomposes the action into ordered sub-actions and generates retrieval strings; CLIP (ViT-bigG-14) scores frame similarity; a Viterbi decoder enforces the fixed sub-action ordering to assign each frame to a sub-action; an LLM then maps sub-actions to relevant candidate differences, yielding temporally-localized frame sets per difference. 3. Action Differencer (VLM): A VLM (GPT-4o) is queried with the localized frames as a three-way multiple-choice VQA: does the difference apply more to video A, video B, or neither?

Two task settings: closed-set (binary A/B prediction given ground-truth difference strings) and open-set (generate differences + predict direction), evaluated with accuracy and recall@N_diff respectively.

Key Contributions

  • Definition of the Video Action Differencing task in closed and open settings.
  • VidDiffBench: 549 video pairs, 4,469 human-annotated fine-grained differences, 2,075 localization timestamp annotations, spanning 5 domains and 18 action categories; expert-validated difference taxonomy with 148 distinct difference descriptions; inter-annotator disagreement rate of only 2%.
  • VidDiff Method: zero-shot agentic pipeline (LLM → CLIP+Viterbi → VLM) that decomposes video differencing into localization and visual comparison.
  • Systematic benchmark of GPT-4o, Gemini-1.5-Pro, Claude 3.5 Sonnet, Qwen2-VL-7B, and LLaVA-Video on VidDiffBench.
  • Ablation isolating contributions of frame localization and Viterbi decoding.

Results

Closed-set accuracy (random = 50%): - Gemini-1.5-Pro: 57.7% avg (best LMM) - VidDiff (ours): 56.3% avg (2nd overall, beats GPT-4o baseline of 53.5%) - GPT-4o: 53.5%; Claude 3.5 Sonnet: 53.4%; LLaVA-Video: 52.3%; Qwen2-VL-7B: 50.4% - On hard split: no model exceeds random chance significantly

Open-set recall@N_diff: - VidDiff: 42.1% avg (best overall; 49.9% on medium) - GPT-4o: 41.7%; Claude 3.5 Sonnet: 35.6%; Gemini: 28.3%; LLaVA-Video: 8.4%; Qwen2-VL-7B: 7.2%

Ablations (easy split, closed-set): - Oracle GT timestamps → VQA: 78.6% (upper bound on visual comparison capability) - Random frame retrieval: 50.1% (chance) - VidDiff without Viterbi: 57.4%; VidDiff with Viterbi: 62.7%

Limitations

  • Relies entirely on general-purpose foundation models; no domain-specific fine-tuning, which limits performance on specialized tasks (surgery, music finger movements).
  • Zero-shot VLMs plateau well below oracle performance even with perfect frame localization (78.6% oracle vs. 62.7% with retrieved frames), indicating fine-grained image differencing itself is a bottleneck.
  • Frame localization via CLIP retrieval fails for precise sub-action alignment needed in hard cases (e.g., diver entry angle, speed of piano scales).
  • Open-set performance depends heavily on LLM difference-recall quality; Gemini's low recall (66% error rate on easy) shows current LMMs miss key domain-relevant differences.
  • Benchmark coverage limited to 5 domains; generalization to other skilled activities (e.g., dance, manufacturing) untested.
  • No training-time adaptation studied; fine-tuning VLMs for fine-grained comparison is explicitly flagged as future work.

Relevance to Agentic AI / LLM Agents

VidDiff is a direct example of task decomposition via agentic workflow — it demonstrates that breaking a complex perception task (cross-video fine-grained comparison) into a propose→localize→verify pipeline achieves measurably better performance than prompting a single LMM end-to-end, reinforcing the value of structured multi-step agent architectures for hard reasoning tasks. The use of heterogeneous specialized components (LLM for proposing/reasoning, CLIP for retrieval, VLM for visual QA) with a deterministic algorithm (Viterbi) as a connector is a concrete instantiation of the tool-augmented agent pattern. For researchers tracking LLM agents in grounded, perception-heavy domains, this work provides both a benchmark (VidDiffBench) and an ablation-validated blueprint for when agentic decomposition adds measurable value over monolithic LMM calls.