Skip to content

Video Action Differencing

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Video Action Differencing (VidDiff) introduces a new task of identifying fine-grained differences between two videos of the same action (e.g., expert vs. novice squat). The authors release VidDiffBench (549 video pairs, 4,469 annotated differences) and propose a three-stage agentic workflow that decomposes the task into proposal, localization, and differencing sub-steps using specialized foundation models. State-of-the-art LMMs fail near-random on this benchmark, while the structured VidDiff method substantially outperforms them.

Problem

Existing video understanding models cannot perform fine-grained, natural-language comparison of two videos of the same action. Prior work either coarsely classifies motion errors, visualizes pose differences, or handles cross-action differencing—none address the sub-action temporal localization + fine-grained frame comparison challenge required for skill-feedback applications (coaching, surgical training, sports analytics).

Method

The VidDiff Method is an agentic three-stage pipeline:

  1. Difference Proposer — an LLM (GPT-4o) generates a set of candidate visually-assessable difference statements given only the action description string \(s\).
  2. Frame Localizer — the same LLM decomposes the action into an ordered sub-action sequence with CLIP retrieval strings; a pretrained CLIP model scores frame–text similarity; a Viterbi algorithm assigns each frame to its sub-action while respecting temporal ordering; the LLM then maps sub-actions to candidate differences, yielding localized frame sets per difference.
  3. Action Differencer — a VLM (e.g., GPT-4o) receives only the localized frames for a given difference and answers a structured multiple-choice question: does the difference apply more to video A, video B, or neither?

For the closed setting, stage 1 is replaced with the ground-truth difference strings, isolating the localization+comparison challenge.

Key Contributions

  • VidDiff task definition — both closed-set (binary attribution given candidate strings) and open-set (generate + attribute) formulations.
  • VidDiffBench — 549 video pairs, 4,469 human-annotated fine-grained differences, 2,075 localization timestamps across 5 domains (fitness, ballsports, surgery, music, diving); first large-scale video-action-differencing dataset.
  • Annotation taxonomy — domain-expert-derived structured difference taxonomies per action to resolve ambiguity and annotator calibration issues.
  • VidDiff Method — zero-shot agentic workflow (LLM + CLIP + VLM) that substantially outperforms end-to-end LMM baselines on the closed-set task.
  • Open-set evaluation metric using GPT-4o partial-matching and recall@\(N_\text{diff}\) (with \(N_\text{diff} = 1.5 \times\) ground-truth count to absorb coverage ambiguity).

Results

  • All tested LMMs (GPT-4o, Gemini-1.5-Pro, Claude 3.5 Sonnet, Qwen2-VL-7B, LLaVA-Video) perform near chance (50%) on the closed-set task, confirming the benchmark difficulty.
  • The VidDiff Method outperforms all LMM baselines on the closed-set evaluation; the Frame Localizer stage is identified via ablation as a key contributor.
  • Specific closed-set accuracy numbers are not fully reproduced in the provided excerpt; the paper states the method "outperforms the baselines, especially in the close-set evaluation."
  • Open-set recall@\(N_\text{diff}\) results mirror the closed-set trend, with structured decomposition benefiting both settings.

Limitations

  • Open-set evaluation relies on a GPT-4o judge for string matching, introducing potential evaluation bias.
  • Frame-level localization uses CLIP with Viterbi, which may fail for non-sequential or overlapping sub-actions.
  • Benchmark is limited to five action domains; generalization to unconstrained in-the-wild video is unverified.
  • The method is compute-heavy (multiple LLM/VLM calls per difference candidate per video pair).
  • Ground-truth annotation requires domain experts per action category, limiting scalability to new domains.

Relevance to Harnesses / Meta-Harnesses

VidDiff is a concrete example of a task-specific agentic harness: rather than prompting a single LMM end-to-end, it decomposes a complex perception task into a sequenced pipeline of specialized sub-agents (LLM proposer → CLIP localizer → VLM differencer), each with a narrowly scoped interface. This pattern—selecting the right model for each sub-task and passing structured intermediate representations between stages—is the core design principle of multi-model meta-harnesses. The paper's explicit citation of Anthropic's agentic workflow framing signals awareness of this design paradigm, and the ablation structure (isolating stage contributions) is a template for evaluating harness component value. For harness researchers, VidDiff is a strong case study in how decomposition overcomes the limits of monolithic LMM inference on temporally-structured, fine-grained perception tasks.