Skip to content

ShotBench: Expert-Level Cinematic Understanding in Vision-Language Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ShotBench is a benchmark of 3,572 expert-annotated QA pairs from 200+ Oscar-nominated films spanning eight cinematography dimensions (shot size, framing, camera angle, lens size, lighting type, lighting condition, composition, camera movement). Evaluating 24 leading VLMs reveals that even GPT-4o tops out at 59.3% accuracy, far below professional proficiency. The authors also release ShotQA (~70k QA pairs) and ShotVL, a fine-tuned model series that surpasses GPT-4o and Qwen2.5-VL-72B using only 3B–7B parameters.

Problem

Existing VLM benchmarks do not assess cinematographic comprehension—the ability to identify and reason about shot size, lens type, camera movement, lighting, and composition at a professional level. This gap hinders both fine-grained visual understanding research and AI-assisted video/image generation that requires cinematographic control signals.

Method

Benchmark construction (ShotBench): Images (3,049) and video clips (464) are collected from films primarily nominated for the Academy Award for Best Cinematography, filtered via LAION aesthetic predictor and NSFW detection, shot-segmented with TransNetV2. Annotators trained on professional cinematography materials label eight dimensions; image labels are bootstrapped from the Shotdeck professional reference platform then verified by expert review. Final multiple-choice QA pairs (4 options, one correct) achieve >90% annotation accuracy through iterative expert audits.

Training dataset (ShotQA): ~70k QA pairs from 58k images and 1.2k video clips across 243 films, formatted identically to ShotBench.

Model (ShotVL): Two-stage training on Qwen2.5-VL-3B/7B-Instruct: 1. SFT on ~60k ShotQA samples with cross-entropy loss to align visual features with cinematic terminology. 2. GRPO on ~8k curated samples using a binary rule-based reward (correct answer = 1, else 0) plus a format reward enforcing <think>...</think><answer>...</answer> structure; group size G=12, clipping ε=0.2, 10 epochs at lr=1e-6.

Key Contributions

  • ShotBench: first benchmark comprehensively covering all eight canonical cinematography dimensions, with 3,572 expert-verified QA pairs from professionally curated films.
  • Systematic evaluation of 24 open-source and proprietary VLMs, quantifying specific failure modes (fine-grained terminology alignment, spatial/camera reasoning, motion perception).
  • ShotQA: first large-scale (∼70k QA) cinematic language dataset.
  • ShotVL (3B and 7B): SFT+GRPO-trained models that surpass both GPT-4o and Qwen2.5-VL-72B on ShotBench at dramatically lower parameter cost.

Results

  • Best baseline on ShotBench: GPT-4o at 59.3% average accuracy; Qwen2.5-VL-72B-Instruct at 59.1%; ~50% of evaluated models fall below 50%.
  • Camera movement is the hardest dimension: >50% of models score below 40%, near random chance (25% for 4-choice).
  • ShotVL-3B: 65.1% average, +19.0 points over baseline Qwen2.5-VL-3B (46.1%), surpassing GPT-4o and Qwen2.5-VL-72B across all eight dimensions.
  • ShotVL-7B: 70.1% average, further gains on every dimension (e.g., Camera Angle: 74.1% vs. GPT-4o's 58.2%; Camera Movement: 60.6% vs. GPT-4o's 48.3%).
  • Ablation: SFT→GRPO outperforms SFT alone (53.5% vs. 54.3% for CoT-SFT, 55.5% for SFT→GRPO at reduced scale); CoT-SFT with Gemini-generated reasoning chains provides negligible benefit over plain SFT, underscoring GRPO's advantage with outcome-only reward.
  • Within model families (InternVL3, Qwen2.5-VL, VILA1.5), larger models consistently outperform smaller ones.

Limitations

  • ShotVL is trained and evaluated on the same distribution (ShotQA → ShotBench); generalization to out-of-distribution cinematographic styles is unverified.
  • Camera movement evaluation relies on video clips; performance on this dimension remains weak even after fine-tuning (ShotVL-3B: 51.7%), indicating fundamental difficulty in temporal motion reasoning.
  • CoT-SFT rationale quality is bounded by Gemini-2.0-flash generation quality, introducing noise; higher-quality reasoning chains were not explored.
  • Benchmark covers only eight dimensions of cinematography; aspects such as color grading, focus/depth-of-field, and editing rhythm are excluded.
  • Annotations from Shotdeck may carry platform-specific labeling biases that propagate into ShotQA.

Relevance to Vision-Language Models

ShotBench provides a hard, domain-specific probe that exposes a concrete ceiling in VLM fine-grained visual perception and spatial reasoning even for frontier models, which is directly relevant to researchers studying VLM evaluation methodology and capability limits. The SFT+GRPO recipe demonstrates that targeted domain fine-tuning with outcome-based RL can close large performance gaps against models 10–24× larger, offering a replicable template for other specialized visual domains. For VLM researchers working on video generation or embodied agents that require cinematographic control, ShotQA and ShotVL provide both a training resource and a baseline. The finding that GRPO implicitly improves camera-movement understanding even when trained only on static images suggests RL-based reasoning augmentation has cross-modal transfer effects worth investigating further.