Skip to content

Self-alignment of Large Video Language Models with Refined Regularized Preference Optimization

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LVLMs suffer from hallucination, poor temporal understanding, and over-reliance on linguistic cues. This paper introduces a self-alignment framework that generates preference pairs by applying spatio-temporal perturbations to elicit model errors, then trains with RRPO — a novel preference optimization method using sub-sequence-level refined rewards and token-wise KL regularization — to outperform DPO and contemporaries across video understanding benchmarks.

Problem

Existing LVLMs struggle with fine-grained temporal reasoning, hallucinate frequently, and ignore visual content in favor of linguistic priors. Prior alignment methods like DPO apply coarse response-level rewards that penalize all tokens uniformly, causing large gradients, model divergence, and imprecise alignment unsuitable for correcting specific erroneous concepts.

Method

Self-alignment data construction: Starting from public video-QA instruction tuning data, the framework applies spatio-temporal perturbations to videos — frame masking (25–50% of pixels) combined with temporal shuffling (random, local, or global) — to provoke the LVLM into generating linguistically-driven incorrect responses. Only incorrect responses are retained as non-preferred samples; an LLM (GPT-4o-mini) identifies the specific incorrect concepts and aligns them with correct counterparts to form concept-paired preference data.

RRPO training objective: Instead of DPO's response-level reward, RRPO computes sub-sequence-level rewards over only the N differing concept spans between preferred (y⁺) and non-preferred (y⁻) responses:

u = Σᵢ [rθ(x, yᵢ⁺) − rθ(x, yᵢ⁻)], with rθ computed as log-probability ratios over token subsequences T[sᵢ:eᵢ].

To prevent reward hacking (producing short, concept-only outputs), RRPO adds a token-wise KL divergence term (DTKL) between the reference model and πθ over the full preferred response:

L_RRPO = −E[log σ(u)] + α·DTKL(x, y⁺)

Theoretically, RRPO's gradient norm is bounded by βM(2NL) versus DPO's βM(|y⁺|+|y⁻|), and DTKL further reduces it, enabling 10× larger learning rates without divergence. Training uses LoRA on the LLM components; vision encoders are frozen.

Key Contributions

  • Self-alignment pipeline that is annotation-free, scalable, and generates model-specific preference pairs via spatio-temporal video perturbation
  • RRPO: sub-sequence-level refined reward targeting only differing concept tokens, with token-wise KL regularization as a trust-region constraint
  • Theoretical proof that RRPO gradient magnitude is strictly smaller than DPO's, explaining empirical stability
  • Validated on three diverse LVLMs (VideoChat2-7B, LLaVA-Video-7B, LongVU-7B) spanning different vision encoders, cross-modal adapters, and LLM backbones

Results

  • vs. DPO/DPA/TDPO/DDPO on LongVU-7B (Table 2): RRPO achieves +2.5%/+5.4% average gain vs. base, versus +0.7–0.9% for all DPO variants; VideoHallucer: 44.0 (RRPO) vs. 40.9 best DPO-variant
  • Model divergence (Figure 5): RRPO KL divergence ≈ 1 at 10× higher learning rate vs. DPO KL divergence ≈ 20; RRPO attains highest performance at lowest divergence
  • vs. off-the-shelf aligned LVLM (Table 3): LLaVA-Video-RRPO outperforms LLaVA-Video-TPO (DPO+SFT with curated data) on all 8 benchmarks; VidHalluc: 55.8 vs. 50.6 (+5.2%)
  • Hallucination (Table 4): RRPO improves VideoHallucer by 4.8–8.8% over base models; gains of up to 10.8% over DPO on VideoHallucer
  • Temporal understanding: Up to +2.8% on TVBench and TempCompass vs. base
  • Long video (MLVU/LongVideoBench): Consistent gains across all three base models
  • Data scaling (Table 6): Performance improves monotonically from 5K to 20K samples; TVBench: 50.9→55.7 for LLaVA-Video

Limitations

  • Static preference dataset; no iterative self-alignment loop (acknowledged as future work)
  • Non-preferred responses are generated via artificial perturbations rather than naturally occurring model failures, which may not fully cover real error distributions
  • Concept identification and response rewriting rely on GPT-4o-mini, introducing API dependency and potential quality variability for open-ended questions
  • Evaluated only at 7B scale; behavior at larger scales unexplored
  • Perturbation type optimal for one architecture differs from others (RS-Mask for VideoChat2, LS-Mask for LLaVA-Video/LongVU), requiring tuning per model

Relevance to Vision-Language Models

This work directly addresses a core failure mode in video-VLMs — the model ignoring visual content in favor of language priors — through a training signal derived from the model's own errors on perturbed inputs, making it a practical self-supervised alignment technique. The RRPO formulation is a principled extension of DPO that is relevant to any VLM fine-grained alignment scenario where preferred/non-preferred responses differ only in specific tokens or concepts. The sub-sequence reward approach and its theoretical analysis of gradient stability are broadly applicable beyond video to any multimodal preference optimization setting. The perturbation-based data construction strategy also connects to work on probing and diagnosing vision-language representational misalignment.