Skip to content

Animation2Code: Evaluating Temporal Visual Reasoning in Video-to-Code Generation

🕒 Published (v1): 2026-06-26 20:38 UTC · Source: Arxiv · link

Why this paper was selected

Temporal visual reasoning evaluation gap; Alane Suhr; tests if VLMs understand dynamics

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Animation2Code introduces a benchmark of 1,069 web animation video-code pairs that tests VLMs on de-rendering videos to executable HTML/CSS/JS code. State-of-the-art VLMs achieve strong appearance similarity (up to 0.84) but consistently fail at temporal dynamics (≤0.31 across all models and settings), revealing a fundamental gap in temporal visual reasoning that persists under finetuning and iterative refinement.

Problem

Prior visual-to-code benchmarks (Design2Code, WebVR, etc.) focus on static de-rendering from images — mapping webpages, charts, or SVGs to code. No benchmark evaluates whether VLMs can recover temporal dynamics from video and translate continuous motion into discrete programmatic representations. Dynamic visual perception is fundamentally harder than static de-rendering because it requires reasoning over object trajectories, timing, and emergent motion, then mapping those to executable code.

Method

Dataset: 1,069 CodePen animations (86% pure CSS, 14% CSS+JS) normalized to self-contained HTML with inline CSS/JS, deterministically rendered to MP4 (1024×768, 30fps) via headless Chromium with frame-stepping. Duration: 2–8s (mean 5.1s). Motion types: rotation (58.9%), translation (52.3%), scaling (35.0%), opacity (25.9%); 65.9% combine ≥3 patterns. 80/20 train/test split.

Metrics disentangle appearance and temporal fidelity: - Appearance similarity: DreamSim embeddings on cropped animation regions, aligned via Dynamic Time Warping (DTW). Mean cosine similarity along the optimal alignment path. - Temporal similarity: CoTracker3 tracklets on a 30×30 grid, computing per-pair displacement correlation (direction cosine similarity + speed magnitude ratio), aggregated via Chamfer distance.

Models evaluated: Gemini 3 Flash Preview (native video at 2/24 FPS), Qwen3-VL-8B-Instruct (video), GPT-5.4 (2 FPS frames), Claude Sonnet 4.6 (frames), LLaMA 4 Scout (frames). Settings: zero-shot, supervised finetuning (LoRA and full SFT on Qwen3-VL), and iterative refinement (METAL multi-agent framework, 3 iterations).

Key Contributions

  • First benchmark for dynamic visual perception via de-rendering — maps video to executable code, requiring complete understanding of both appearance and motion.
  • Human-aligned evaluation suite — appearance and temporal similarity metrics validated via ROC-AUC against 600 human pairwise annotations (3–4 annotators each, Krippendorff's \(\alpha = 0.81\) overall).
  • Consistent temporal gap — all SOTA VLMs achieve \(\leq 0.31\) temporal similarity despite near-saturated execution success (\(\geq 97\%\)) and strong appearance (up to 0.84).
  • SFT degrades quality — finetuned Qwen3-VL learns code priors over video grounding, improving exec success but dropping appearance from 0.67→0.46 and temporal from 0.23→0.08.

Results

  • Best zero-shot: GPT-5.4 achieves appearance 0.84, temporal 0.29 (exec 100%). Claude Sonnet 4.6: appearance 0.82, temporal 0.29.
  • Overall temporal ceiling: Gemini 3 Flash Preview (24 FPS native video) reaches the highest temporal at just 0.31.
  • SFT (Qwen3-VL-8B): LoRA exec 98.6% but appearance 0.43 / temporal 0.09; Full SFT exec 94.9% but appearance 0.46 / temporal 0.08.
  • Iterative refinement (Qwen3-VL, 3 iters): improves appearance from 0.67→0.73 (+4.4% first iter, +0.9% last) and temporal from 0.23→0.28 (+9.0% first iter, +3.0% last).
  • FPS scaling: 24 vs 2 FPS yields only marginal gains (Gemini: 0.80→0.80 appearance, 0.31→0.30 temporal).
  • Static rate: GPT lowest (13.1%); Qwen highest (41.5% of executable outputs are static).
  • Human correlation: Combined logistic regression (appearance + temporal deltas) achieves 83.9% accuracy predicting overall human preference; our metrics outperform VLM-as-a-judge in every dimension.

Limitations

  • Domain specificity — dataset sourced exclusively from CodePen (MIT-licensed web animations), not real-world video.
  • Small model sweep — only 5 models tested; SFT limited to Qwen3-VL-8B-Instruct.
  • Noisier temporal annotations — inter-annotator agreement for temporal judgment (\(\alpha = 0.73\)) is lower than appearance (\(\alpha = 0.81\)), suggesting motion quality is inherently harder to judge.
  • Marginal benefit from richer input — neither higher FPS nor native video input closes the temporal gap, indicating the bottleneck is model reasoning rather than input resolution.

Relevance to Vision-Language Models

Animation2Code pinpoints a clean, measurable failure mode of current VLMs: they can reconstruct what is visible but not how it moves. The systematic gap between appearance and temporal scores (often 0.50+ difference) provides a targeted benchmark for the field to track progress on temporal visual reasoning. The disentangled metrics are especially valuable — they isolate motion understanding from visual fidelity, making it possible to attribute gains to the right capability.