DenseDPO: Fine-Grained Temporal Preference Optimization for Video Diffusion Models

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DenseDPO is a preference optimization framework for text-to-video diffusion models that generates structurally aligned video pairs via guided denoising of real videos, then collects segment-level (1s) preference labels rather than single binary labels per clip. This eliminates the motion bias in vanilla DPO (annotators favoring low-motion, artifact-free videos) and yields a denser supervision signal, enabling dynamic video generation with one-third the labeled data. A key finding is that off-the-shelf VLMs (GPT-o3) can reliably annotate short-segment preferences, enabling scalable automatic labeling without task-specific fine-tuning.

Problem

Vanilla DPO for video diffusion generates preference pairs from independent noise seeds, producing clips with disparate motion patterns. Annotators systematically prefer artifact-free slow-motion videos over artifact-prone dynamic ones, creating a motion bias that suppresses dynamic degree after DPO training. Additionally, whole-video preference labels are coarse: artifacts appearing at different timestamps create inconsistent signals, and existing VLMs fail to reliably assess long (5s) videos.

Method

Guided pair generation (StructuralDPO): Given a real video x and guidance level η ∈ [0.65, 0.8], two corrupted noisy latents are created as xⁿ = (1−η)x + ηϵ with independent noise samples, then denoised from step n = round(η·N) to step 1. The shared structure anchors motion trajectories while allowing local visual variation, neutralizing motion bias.

Segment-level dense labeling (DenseDPO): Videos are split into 1s segments; annotators label each segment pair independently, yielding a label vector l ∈ {−1,+1}^F. The DenseDPO objective sums segment-level implicit reward differences, applying the Bradley-Terry-based Flow-DPO loss per segment. Over 60% of pairs have mixed winning/losing segments across time, which would be lost or mislabeled under a global binary label.

VLM automatic annotation: Temporally aligned 1s segment pairs are fed to GPT-o3 for pairwise preference prediction. Segment results are aggregated by majority voting to predict long-video preferences.

Key Contributions

• Guided video generation strategy that produces structurally similar pairs, eliminating motion bias without requiring additional annotation data.
• Segment-level dense preference labeling formulated as a modified DPO objective that localizes supervision to temporal windows with clear preference signal.
• Demonstration that off-the-shelf VLMs (GPT-o3) achieve 70.59% accuracy on long-video preference via segment aggregation, matching task-specific fine-tuned models; DenseDPO trained on GPT-o3 segment labels nearly matches human-label performance.
• Data efficiency: DenseDPO matches VanillaDPO across all metrics (text alignment, visual quality, temporal consistency) while significantly exceeding it in dynamic degree, using only 10k vs. 30k labeled pairs.

Results

• VideoJAM-bench: DenseDPO VisionReward Dynamic Degree = 0.680 vs. VanillaDPO 0.535 (pre-trained: 0.680); DenseDPO matches VanillaDPO on text alignment (0.863 vs. 0.867), visual quality (0.376 vs. 0.371), temporal consistency (0.632 vs. 0.636).
• MotionBench: DenseDPO Dynamic Degree = 0.858 vs. VanillaDPO 0.709; dynamic degree recovered to near pre-trained (0.840) while improving all other metrics over pre-trained.
• Human evaluation (VideoJAM-bench): DenseDPO vs. VanillaDPO: dynamic degree preference 63.9% for DenseDPO, 12.6% VanillaDPO wins; other dimensions (TA, VQ, TC) near parity.
• VLM labeling accuracy (short 1s segments): VisionReward 72.45%, GPT-o3 70.03%; on 5s long videos, all models degrade (VisionReward 62.11%, GPT-o3 53.45%); GPT-o3 Segment aggregation recovers to 70.59% on long video.
• VLM-label DPO: GPT-o3 Segment + DenseDPO: TA=0.842, VQ=0.368, TC=0.598, DD=0.672 vs. human label: 0.863/0.376/0.632/0.680 (5.5× more VLM-labeled videos used).
• Segment length ablation (5k videos): s=1 outperforms s=2; s=0.5 comparable to s=1 but 2× annotation cost.

Limitations

• VLM-labeled DenseDPO requires 5.5× more videos than human-labeled version to approach comparable performance, due to label noise.
• GPT-o3 Segment labels still underperform human labels on temporal consistency and visual quality metrics.
• Guided generation reduces diversity of training pairs; the diversity–quality tradeoff requires careful tuning of η.
• Segment granularity below 0.5s is impractical: too short for VLMs to assess temporal quality.
• Ablation showing that label quality matters more than quantity suggests noisy VLM labelers may hit a ceiling without model improvements.

Relevance to Vision-Language Models

DenseDPO directly demonstrates a practical use case for VLMs as scalable reward/preference annotators: rather than fine-tuning video reward models on large human-preference corpora, GPT-o3 applied at the segment level achieves competitive labeling accuracy for video DPO. This validates a paradigm where VLMs serve as zero-shot or near-zero-shot judges for generative model alignment, with temporal decomposition as the key enabler — a technique transferable to other long-context multimodal alignment tasks. The finding that existing VLMs fail on 5s video but succeed on 1s clips is an important practical constraint for VLM-as-judge pipelines in video domains. The work also shows that scaling VLM-generated labels (10k → 55k) consistently improves DPO results, pointing to data scaling with stronger VLMs as a near-term path to further gains.