Building a Precise Video Language with Human-AI Oversight

🕒 Published (v1): 2026-04-22 09:01 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026 scalable video-language human-AI oversight; datasets and benchmarks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces a recipe for high-quality video captioning comprising: (1) a structured specification of video language with 200+ visual/motion primitives developed with professional filmmakers, (2) CHAI (Critique-based Human–AI Oversight), a scalable annotation framework where experts critique model-generated captions rather than write from scratch, and (3) post-training strategies (SFT, DPO, inference-time scaling) using the resulting (pre-caption, critique, post-caption) triplets. Fine-tuning Qwen3-VL on ~15k such triplets surpasses closed-source Gemini-3.1-Pro on the released benchmark.

Problem

Existing video-text datasets (MSR-VTT, ActivityNet, ShareGPT4Video, UltraVideo, etc.) lack formal annotation policies, producing captions that (1) misuse cinematography terms (e.g., zoom vs. dolly, aerial vs. bird's-eye), (2) omit key visual elements (camera shake, focal shifts, POV), and (3) include subjective or emotional language irrelevant to observable content. Without precise supervision signals, video-language models hallucinate objects/motion and fail to model fine-grained camera and spatial dynamics.

Method

Specification. In collaboration with 100+ professional content creators (filmmakers, cinematographers, game designers) over one year, the authors formalize a structured captioning policy spanning five aspects: subject, scene, motion, spatial, and camera. The taxonomy—CameraBench-Pro—defines 200+ primitives (~50 camera motion, ~100 camera setup, ~70 cinematography), each with rigorous definitions, video examples, corner cases, and decision rules. Static properties (focal plane, shot size) are labeled at both shot start and end to capture temporal change.

CHAI Oversight Framework. A five-step human–AI pipeline: 1. Annotators label visual/motion primitives on a separate platform (reduces cognitive load during captioning). 2. A VLM generates a high-recall pre-caption from the primitive labels following the captioning policy. 3. Trained human experts write a correctional critique enumerating what is wrong, missing, or out of order and how to fix it. 4. The VLM incorporates the critique to produce a post-caption. 5. Humans revise the critique if the post-caption still contains errors; a separate reviewer role audits every critique and post-caption with accuracy-based bonuses enforcing precision.

Annotators are recruited from content creators, filtered through six rounds of 150-video exams; only the top 3% of 600+ applicants proceed to a month of paid training. Shadowing of 100 gold-standard captions authored by the paper's authors follows before live annotation.

Post-Training. The (pre-caption, critique, post-caption) triplets provide three training signals for Qwen3-VL: supervised fine-tuning (SFT) on post-captions, DPO on preference pairs (post > pre), and critique supervision for reward modeling and inference-time scaling via a critique-then-caption chain.

Key Contributions

• Structured video captioning specification with 200+ primitives developed with professional creators; released publicly.
• CHAI: a scalable human–AI annotation framework that shifts human effort from generation to verification, improving accuracy, completeness, and fluency simultaneously.
• ~20k (pre-caption, critique, post-caption) triplets across films, games, commercials, and user-generated clips; 5k held-out as a benchmark.
• First unified benchmark jointly evaluating (1) caption generation, (2) reward modeling, and (3) critique generation on the same videos.
• Post-training recipe showing that open-source Qwen3-VL fine-tuned with modest expert supervision outperforms closed-source Gemini-3.1-Pro.
• Re-captioning pipeline applied to large-scale professional footage, enabling fine-tuned Wan video generation to follow 400-word prompts with professional-level control over camera motion, angle, lens, focus, and framing.

Results

• Fine-tuned Qwen3-VL outperforms Gemini-3.1-Pro on the released benchmark across caption generation, reward modeling, and critique generation tasks (specific BLEU-4 and accuracy numbers are tabulated in Table 1 and Table 9, not fully reproduced in the provided excerpt).
• Reward modeling is evaluated as binary accuracy (chance = 0.5) predicting post-caption > pre-caption.
• Benchmark scale: ~5k held-out triplets, far exceeding prior public benchmarks (DREAM-1K: ~1k; TUNA-Bench: ~1k).
• Ablations show that removing any of precision, recall, or constructiveness from critiques substantially degrades post-training performance.
• VLMs consistently score well on subject appearance and scene context but poorly on subject motion and camera dynamics off-the-shelf.
• Wan fine-tuned on re-captioned professional footage achieves finer cinematographic control including camera motion, angle, lens, focus, POV, and framing with prompts up to 400 words.

Limitations

• Annotation pipeline is highly resource-intensive: only 3% of 600+ applicants qualify, requiring professional content-creation backgrounds; not easily reproducible at arbitrary scale without significant infrastructure.
• Benchmark covers ~4k evaluation videos; generalization to domains beyond films, games, and commercials is unverified.
• Post-training gains depend critically on critique quality, which itself requires the same expensive human oversight; automating critique generation without quality degradation remains an open problem.
• Specific numeric results (BLEU-4, reward model accuracy) are referenced but not fully available in the provided text excerpt, limiting direct comparison replication.
• Focus is on short professional clips; how the specification and CHAI scale to long-form video (multi-minute) is not addressed.

Relevance to Vision-Language Models

This paper directly tackles the data bottleneck for video-VLMs by showing that annotation policy and oversight framework matter as much as scale—a lesson parallel to findings in image-VLM data curation (e.g., LLaVA, BLIP-2 recaptioning pipelines) but extended to the temporally dynamic video domain. The CHAI triplet structure is a concrete instantiation of scalable oversight for VLMs, enabling preference-based post-training (DPO) and reward modeling from human-verified feedback rather than synthetic signals. For practitioners training or fine-tuning video-VLMs, the released specification, benchmark, and Qwen3-VL checkpoints provide a direct resource; the finding that critique quality (precision, recall, constructiveness) governs downstream VLM performance is an actionable design principle. The downstream application to text-to-video generation further demonstrates how improved video-VLM captioning closes the loop to conditional generation quality.