Skip to content

Video Action Differencing

🕒 Published (v1): 2025-03-10 21:18 UTC · Source: Arxiv · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VidDiff introduces the task of identifying fine-grained natural-language differences between two videos of the same action (e.g., expert vs. novice squat). The authors release VidDiffBench (549 video pairs, 4,469 annotated differences across fitness, sports, surgery, music, and diving) and propose the VidDiff Method, a three-stage agentic workflow that decomposes the task into difference proposal, keyframe localization, and frame-level visual comparison. State-of-the-art LMMs struggle on this benchmark; the structured agentic pipeline outperforms direct LMM baselines.

Problem

Existing video understanding research either performs coarse-grained action comparison or relies on domain-specific training data. No large-scale benchmark or general-purpose method exists for fine-grained, zero-shot natural-language identification of subtle within-action differences — a capability needed for coaching, surgical training, and skill acquisition.

Method

The VidDiff Method is an agentic three-stage pipeline:

  1. Difference Proposer — an LLM generates candidate visual difference statements given only the action description string \(s\) (open setting) or uses ground-truth difference strings (closed setting).
  2. Frame Localizer — an LLM decomposes the action into sub-actions and generates CLIP retrieval strings; a pretrained CLIP model scores frame–text similarity, and a Viterbi-based algorithm assigns each frame to its sub-action in sequence-order. The LLM then maps sub-actions to candidate differences, yielding temporally localized frame sets per difference.
  3. Action Differencer — a VLM receives the localized frames from both videos and answers a structured multiple-choice question: does this difference apply more to video A, video B, or neither?

Closed setting: binary A/B accuracy (chance = 50%).
Open setting: recall@\(N_\text{diff}\) where \(N_\text{diff} = 1.5 \times |\text{labeled differences}|\); GPT-4o used for partial-match string alignment between predicted and ground-truth difference descriptions.

Key Contributions

  • Novel Video Action Differencing (VidDiff) task definition with both closed-set (discrimination) and open-set (generation + discrimination) variants.
  • VidDiffBench: 549 video pairs, 4,469 fine-grained difference annotations, 2,075 localization timestamps across 5 domains (fitness, ballsports, diving, music, surgery), with expert-derived annotation taxonomies to reduce ambiguity and calibration variance.
  • VidDiff Method: zero-shot agentic workflow combining LLM proposal, CLIP+Viterbi temporal localization, and VLM frame comparison.
  • Systematic LMM benchmarking (GPT-4o, Gemini-1.5-Pro, Claude 3.5 Sonnet, Qwen2-VL-7B, LLaVA-Video) establishing difficulty baselines.
  • Ablation studies isolating the contribution of temporal localization to overall pipeline performance.

Results

  • All evaluated LMMs perform near or only modestly above chance (50%) on the closed-set task, demonstrating the benchmark's difficulty.
  • The VidDiff Method outperforms direct LMM baselines on closed-set evaluation.
  • Ablations confirm that the Frame Localizer stage (CLIP+Viterbi sub-action segmentation) is a key contributor to performance gains over end-to-end LMM prompting.
  • Open-set recall@\(N_\text{diff}\) remains low for all models, underscoring the challenge of simultaneous difference generation and localization.
  • (Note: specific numerical accuracy figures are not fully reproduced in the provided text excerpt.)

Limitations

  • Benchmark evaluation for the open setting relies on GPT-4o as a judge for string matching, introducing potential LLM-judge bias.
  • Localization annotations cover only a subset of video pairs (2,075 timestamps for 4,469 differences), limiting granular localization ablations.
  • Frame sampling strategy (2–6 fps) may discard critical sub-second events, particularly in fast actions like ballsports or diving.
  • CLIP-based localization is text–frame similarity only; it does not model inter-video alignment or temporal correspondences across the two videos jointly.
  • Annotation taxonomy was constructed per-action by domain experts, limiting extensibility to new action categories without expert input.
  • Open-set generation quality depends on the Proposer LLM's world-knowledge priors, which may be miscalibrated for niche domains (e.g., surgical knot tying).

Relevance to Harnesses / Meta-Harnesses

VidDiff Method is a textbook task-decomposition harness: it wraps heterogeneous specialist models (LLM, CLIP, VLM) in a fixed orchestration pipeline where each stage's output is the structured input to the next, directly mirroring the meta-harness pattern of routing subtasks to best-fit foundation models. The paper explicitly cites Anthropic's agentic workflow framework as the design paradigm, making it a concrete applied example of a perception-oriented harness in a zero-shot setting. For harness researchers, it illustrates how Viterbi-constrained retrieval can serve as a deterministic "glue" layer between neural components, and how structured multiple-choice reformulation at the final stage reduces the VLM's output space — a recurring harness design principle. The benchmark's ablation structure (localization on/off) also provides a methodology for isolating the contribution of individual harness stages.