Skip to content

Stable Cinemetrics : Structured Taxonomy and Evaluation for Professional Video Generation

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Stable Cinemetrics (SCINE) introduces a structured evaluation framework for professional video generation built around four hierarchical cinematic taxonomies (Setup, Events, Lighting, Camera) covering 76 fine-grained control nodes. A large-scale human study with 84 film professionals annotates 20K videos from 13+ T2V models. A VLM fine-tuned on these annotations achieves 72.36% alignment accuracy, outperforming zero-shot baselines by ~20%.

Problem

Existing T2V benchmarks (VBench, VideoPhy, T2V-CompBench) evaluate general video quality but lack the shot-level cinematic specificity required for professional use—they omit controls like camera framing, lighting position, and narrative pacing that are fundamental to filmmaking. There is no standardized taxonomy or evaluation protocol for assessing professional cinematic control in generative video.

Method

SCINE constructs four hierarchical taxonomies grounded in industry practice: Setup (scene, subjects, text generation), Camera (intrinsics, extrinsics, trajectory, creative intent), Lighting (source, color temperature, conditions, position, effects), and Events (actions, emotions, dialogues with temporal/contextual decomposition). From these, two prompt types are generated: SCINE-Scripts (narrative/screenplay-style, seeded by a professional screenwriter then LLM-expanded using Events nodes) and SCINE-Visuals (SCINE-Scripts enriched with sampled Camera/Lighting/Setup nodes). Taxonomy-guided prompt generation enables automated per-control categorization and targeted question generation for isolated evaluation of each of 76 leaf nodes. Human annotation (84 experts, 1–5 scale, 3 votes/pair, 248K total annotations) covers 13 T2V models on 2,089 prompts. For scalable evaluation, Qwen2.5-VL-7B is fine-tuned on 44K binary preference samples (human annotation aggregated via Bradley–Terry objective) to predict pairwise preferences per taxonomy node.

Key Contributions

  • Four hierarchical cinematic taxonomies with 76 leaf-level control nodes grounded in professional filmmaking practice.
  • SCINE benchmark: 2,089 prompts (Scripts + Visuals) with automated prompt categorization and targeted per-control question generation.
  • Largest professional video evaluation study: 80+ film-domain annotators, 13 T2V models, 20K videos, 248K annotations; ICC of 80.4% at model-pair level.
  • Fine-grained analysis showing current T2V models struggle most with Events and Camera; Setup and Lighting are comparatively easier.
  • Fine-tuned VLM evaluator (7B) aligned to expert preferences, achieving 72.36% pairwise preference accuracy—a ~20% absolute gain over zero-shot Qwen2.5-VL-7B.

Results

  • Top models: WAN-14B and Minimax lead across all taxonomy pillars; LTX-Video consistently underperforms.
  • Hardest pillar: Events (narrative accuracy), then Camera; Setup yields highest absolute scores.
  • Genre: Biography is easiest; Comedy is hardest across all models.
  • Fine-grained camera: Extrinsics and Trajectory score lowest; Creative Intent and Advanced Controls are weakest and show largest drops from basic→advanced prompts.
  • Fine-grained lighting: Lighting Position is the primary bottleneck; Lighting Source shows the largest drop on advanced prompts.
  • Fine-grained subjects: Hair/Accessories best; Personality/Makeup worst.
  • Director role (all controls jointly): Induces performance drop on all models vs. standalone pillar evaluation; Camera degrades most.
  • VLM evaluator: Fine-tuned 7B model at 72.36% vs. zero-shot Qwen2.5-VL-7B (~52%), Qwen2.5-VL-32B, and 72B; scaling zero-shot VLM size yields negligible improvement.
  • Wilcoxon tests: Statistically significant model preferences in 37 of 45 pairwise comparisons.

Limitations

  • Taxonomy developed with a specific expert network; filmmaking terminology varies across cultures, limiting global coverage.
  • Some nodes (Color Temperature, ISO) were abstracted because annotators could not consistently perceive fine-grained numerical values.
  • Prompt generation relies on proprietary LLMs, introducing potential language/style biases.
  • Zero-shot VLM experiments constrained by compute; fine-tuned evaluator assessed only at 7B scale with limited ablation.

Relevance to Vision-Language Models

SCINE directly advances VLM-based video evaluation: the fine-tuned Qwen2.5-VL-7B evaluator demonstrates that expert-annotated, domain-specific training data substantially improves VLM alignment with professional human judgment, underscoring the limits of zero-shot VLM evaluation for fine-grained cinematic understanding. The benchmark provides a structured testbed for measuring whether VLMs can assess shot-level cinematic properties—an open capability gap relevant to video understanding, grounding, and multimodal reasoning. The finding that VLM scale alone does not improve alignment with expert annotations motivates fine-tuning and domain-aligned preference learning as necessary complements to model scaling in VLM evaluation pipelines.