EvalVerse: Pipeline-Aware and Expert-Calibrated Benchmarking for Professional Cinematic Video Generation¶

🕒 Published (v1): 2026-05-22 00:00 UTC · Source: HuggingFace · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

EvalVerse is a pipeline-aware, expert-calibrated evaluation framework for professional cinematic video generation that shifts assessment from "whether it is right" (prompt-following) to "whether it is good" (cinematic quality, acting, aesthetics). It distills judgment from 34 professional experts into a VLM via Chain-of-Thought fine-tuning, enabling scalable machine evaluation aligned with human aesthetic perception. Beyond static benchmarking, it is designed as foundational infrastructure for RL reward models and agentic video generation workflows.

Problem¶

Existing video generation benchmarks (VBench, VBench 2.0, VBench++, UniVBench, EvalCrafter) are trapped in an "evaluation paradox": they measure prompt-following ("rightness") but ignore cinematic "goodness" (acting, affectivity, multi-shot logic, audio-visual synchronization). Generic VLMs used as automated judges lack domain-specific expertise, creating a credibility gap between human aesthetic perception and machine scoring. Additionally, no prior benchmark covers full-modality generation (text-to-video, reference-to-video, multi-shot sequences, synchronized audio) in a single framework.

Method¶

EvalVerse proceeds through five steps:

Taxonomy Establishment: A hierarchical, pipeline-aware taxonomy mirrors the professional filmmaking workflow—3 production stages (Pre-Production, Production, Post-Production) → 7 cinematic aspects → 18 main dimensions → 45 sub-dimensions → 196 granular rationales. Pre-Production covers Visual Concept Design; Production covers Acting, Cinematography, Aesthetics, and Affectivity; Post-Production covers Multi-Shot logic and Sound Design.
Dataset Curation via "Real-to-Gen" Data Engine: Test pairs are constructed by proportional, diversified sampling from a million-scale professional film/television database. Hierarchical structural annotation and asset disentanglement (face, expression, camera trajectory, material, lighting, lip-sync extractors) produce high-fidelity test pairs with authentic reference videos, avoiding stochastic prompt-based bias.
Expert Human Evaluation: 34 professionals (14 AIGC scientists/engineers + 20 professional artists) perform ranking-based scoring watching all candidate videos simultaneously, with approximately 2 hours per pair per expert, plus cross-checking and calibration.
Expert-Calibrated VLM Evaluator: Specialized perception extractors feed structured signals into a VLM (e.g., Gemini, Qwen-VL) that is fine-tuned on both preference-pair data and expert scoring data. The VLM is forced to generate professional-grade Chain-of-Thought rationales before issuing scores, aligning its internal reasoning logic with professional critics.
Applications: The resulting evaluator serves as a benchmarking suite, a diagnostic tool, a reward model for RLHF/GRPO RL training of video models, and an expert evaluator agent inside autonomous video generation pipelines.

Key Contributions¶

First pipeline-aware cinematic evaluation taxonomy decomposing video assessment into 3 stages, 7 aspects, 18 dimensions, 45 sub-dimensions, and 196 rationales
Expert-calibrated CoT fine-tuning pipeline aligning VLM reasoning to 34 professional experts, producing interpretable per-dimension scores
"Real-to-Gen" data engine constructing test pairs from a million-scale professional video database with hierarchical annotation and asset disentanglement
First benchmark with full-modality coverage: text-to-video, reference-to-video, multi-shot narrative sequencing, and audio-visual synchronization simultaneously
Framework explicitly designed to serve as reward model for RL training (RLHF, GRPO) and as evaluator agent within agentic video generation workflows

Results¶

The provided text does not report quantitative correlation or benchmark numbers. Comparative coverage is demonstrated qualitatively: - EvalVerse is the only framework achieving all four modality types (text-to-video ✓, reference-to-video ✓, video-with-sound ✓, multi-shot ✓) - It is the only framework that is pipeline-aware, expert-guided, and produces high-interpretability CoT outputs; all eight prior benchmarks fail on at least one of these axes (Table 1) - Across 14 evaluation dimensions in Table 2, EvalVerse achieves full coverage; no prior benchmark covers more than ~6 of 14 dimensions, and no prior work covers Multi-Shot or Sound dimensions at all

Limitations¶

Post-production scope is restricted to natively generated multi-shot sequences and synthesized audio; traditional editing (cuts, color grading applied post-hoc) and dubbing pipelines are excluded
No quantitative human-machine correlation scores are reported in the available text, making it difficult to assess how well the expert-calibrated VLM actually tracks expert judgments numerically
Expert annotation throughput is ~2 hours per pair per expert; scalability of the human calibration loop is not addressed
The taxonomy is fixed at the time of framework release; adapting it to new modalities (e.g., 3D, interactive video) would require re-annotation

Relevance to Harnesses / Meta-Harnesses¶

EvalVerse is architecturally a meta-harness: it orchestrates a battery of specialized extractors (face, expression, camera trajectory, lip-sync, lighting, material, sentiment) as tool-call modules feeding a central VLM reasoner, with a Planner→Tool Call→Reflection loop explicitly shown in Figure 1's "Agent System" application. The framework is designed with dual downstream harness roles in mind—as a reward signal harness for RL training loops (RLHF/GRPO) and as an evaluator agent embedded inside agentic video generation pipelines. This makes it a direct example of an evaluation harness functioning as a component within larger orchestration stacks, showing how domain-specific expert calibration can be systematized into reusable, composable scoring modules rather than monolithic human review.