Inference-Time Text-to-Video Alignment with Diffusion Latent Beam Search

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Text-to-video diffusion models frequently produce unnatural motion, reverse playback, and static scenes; aligning their outputs to human preference at inference time without retraining remains hard. This paper proposes Diffusion Latent Beam Search (DLBS) with a lookahead estimator, an inference-time search over the reverse diffusion process that selects latent paths maximizing a calibrated reward. It also shows that no single existing video quality metric correlates well with VLM/human preference, necessitating a prompt-dynamics-aware weighted calibration.

Problem

Inference-time alignment of text-to-video diffusion models is undermined by two compounding errors: (1) greedy search and best-of-N sampling use a one-step posterior mean (Tweedie's formula) as the reward input, which is noisy at early denoising steps, causing inaccurate reward signals; (2) individual video quality metrics (subject consistency, motion smoothness, dynamic degree, etc.) have weak and inconsistent Pearson correlation with VLM/human preference scores, so optimizing any single metric leads to over-optimization artifacts.

Method

Diffusion Latent Beam Search (DLBS): At each denoising step \(t\), each of \(B\) latent beams generates \(K\) stochastic DDIM candidates. The one-step posterior mean \(\hat{z}_{0|t-1}\) is computed for each candidate and scored by a proxy reward. The top-\(B\) candidates advance as the new beams; final output is the highest-reward \(z_0\) at \(t=0\). This avoids the greedy argmax error while staying gradient-free.

Lookahead (LA) Estimator: Instead of Tweedie's noisy one-step estimate, each candidate latent \(z_{t-1}\) is denoised for \(T' \ll T\) additional steps via deterministic DDIM (interpolating time steps equally from \(t-1\) to \(0\)), yielding a less-noisy clean estimate \(\tilde{z}_{0|\tilde{t}(0)}\). This tightens the upper bound on reward-approximation error monotonically in \(T'\); empirically \(T'=2\)–\(6\) suffices.

Reward Calibration: Six base metrics (subject consistency via DINO, motion smoothness via AMT, dynamic degree via optical flow, aesthetic quality via LAION predictor, imaging quality via MUSIQ, text-video consistency via ViCLIP) are combined as a weighted linear combination \(r^* = \sum_i w_i r_i / \sum_i w_i\). Weights \(w_i \in \{0, 0.25, 0.5, 0.75, 1.0\}\) are selected by brute-force search to maximize Pearson correlation with Gemini-1.5 scores on prompt sets stratified by dynamics grade (DEVIL-high/medium/static, MSRVTT-test). Dynamic prompts upweight dynamic degree; static prompts upweight subject consistency and motion smoothness.

Key Contributions

• Diffusion Latent Beam Search (DLBS): beam search over the reverse diffusion trajectory as a zeroth-order, gradient-free inference-time alignment method
• Lookahead estimator: \(T'\)-step deterministic DDIM to reduce noisy reward evaluation at intermediate denoising steps, with provable monotonic error-bound tightening
• Reward calibration: dynamics-grade-conditioned weighted linear combination of six perceptual metrics, demonstrating that single-metric optimization is insufficient
• Empirical guideline: allocating inference compute to the LA estimator and larger search budget KB outperforms increasing diffusion steps T
• Demonstration that DLBS is complementary to fine-tuning (VideoDPO)

Results

• vs. BoN and GS (Latte, MSRVTT-test): DLBS-LA (KB=8, T′=6, NFE≈2500) achieves higher Gemini calibrated reward than BoN (KB=64, NFE=3200) — better result at ~22% fewer function evaluations
• Human pairwise win rate: DLBS-LA wins consistently across all models (Latte, CogVideoX-5B, Wan 2.1-14B) and prompt sets vs. BoN (KB=64)
• VideoScore (human-judgment metric, not used in calibration): DLBS substantially improves VideoScore over BoN baseline across all four prompt sets
• Calibrated reward vs. budget scaling: DLBS is the only method that continues improving as KB scales to 32+; BoN and GS saturate or regress
• SoTA models: DLBS-LA improves calibrated reward for CogVideoX-5B and Wan 2.1-14B on DEVIL-very-high and MovieGen Video Bench (1,003 prompts)
• With fine-tuning: VideoDPO alone yields negligible gain; DPO + DLBS achieves notable improvement on DEVIL-high (0.337→0.359) and MSRVTT-test (0.555→0.576)
• Optimal K/B ratio: peak performance at K ≈ 25–50% of budget KB (e.g., K=4–8 at KB=16–32)
• Diversity: DLBS and DLBS-LA maintain higher ViCLIP mean pairwise distance than BoN or GS at equal budget

Limitations

• Reward calibration coefficients are prompt-dynamics-specific; weights must be selected per prompt category, limiting plug-and-play generality
• VLM (Gemini/GPT-4o) feedback is used as a proxy oracle for human preference; calibration quality depends on VLM accuracy
• Lookahead estimator adds denoising calls; most cost is in VAE decode for reward evaluation, but overall NFE increases with T′
• Physical plausibility is not explicitly modeled; authors note that incorporating physics-aware reward models is future work
• Orthogonal to sampling speed improvements (distillation, parallel sampling); combining them is left unaddressed

Relevance to Vision-Language Models

VLMs (Gemini-1.5, GPT-4o) play a dual role here: as the alignment oracle whose preferences define the calibration target, and as the evaluator validating that the calibrated reward transfers to actual quality gains. The finding that no single automated video metric correlates reliably with VLM preference underscores the importance of VLMs as preference models in generative pipelines. The calibrated reward framework — using VLM feedback to supervise the combination of low-level metrics — is a practical template for preference-guided search in any modality where VLM-as-judge is employed. For researchers tracking VLMs, this paper positions VLMs as a scalable substitute for human raters in reward design for video generation alignment.