Skip to content

CHiP: Cross-modal Hierarchical Direct Preference Optimization for Multimodal LLMs

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CHiP extends Direct Preference Optimization (DPO) for multimodal LLMs by adding a Visual Preference Optimization module and a Hierarchical Textual Preference Optimization module that operates at response, segment, and token levels. The method addresses the failure of standard multimodal DPO to align image–text representations and to separate hallucinated from non-hallucinated descriptions. CHiP reduces hallucination rates substantially over DPO and surpasses GPT-4V on object hallucination benchmarks.

Problem

Multimodal DPO—naively extending text-based DPO with image-conditioned preference pairs—fails to (1) bring image and ground-truth text representations close in the LLM's embedding space, and (2) push hallucinated descriptions away from ground-truth descriptions. Response-level preference optimization is also too coarse: it assigns a single reward to an entire generation, making it hard to attribute credit to specific hallucinated spans and increasing susceptibility to reward hacking.

Method

CHiP combines two modules trained jointly via: $\(L_{\text{CHiP}} = L_{\text{DPO}_v} + L_{\text{DPO}_r} + \lambda L_{\text{DPO}_s} + \gamma L_{\text{PO}_k}\)$

Hierarchical Textual Preference Optimization (HDPO): - Response-level (L_DPO_r): standard multimodal DPO objective over full response. - Segment-level (L_DPO_s): up-weights tokens belonging to segments that differ between chosen and rejected responses (identified as hallucinated spans absent in the accepted response), with normalization to prevent length bias (weight λ=1 for Muffin, λ=3 for LLaVA). - Token-level (L_PO_k): minimizes sequential KL divergence token-by-token between the reference and policy, with a stop-gradient on the chosen side; weight γ=0.1.

Visual Preference Optimization (L_DPO_v): Constructs image preference pairs (m_w, m_l) where m_l is a corrupted version of m_w (forward diffusion process at T=500 noise steps). The objective maximizes σ(r(x, m_w, y_w) − r(x, m_l, y_w)), forcing the model to use visual content rather than relying solely on language priors.

Training uses 5k samples from RLHF-V-Dataset, β=0.5, 3 epochs, LR 5e-7, batch 32, on 4 H100 GPUs (~3h for LLaVA-1.6 7B, ~5h for Muffin 13B).

Key Contributions

  • Diagnostic showing via PCA of LLM last-token embeddings that multimodal DPO fails to achieve cross-modal semantic alignment and hallucination separation.
  • Visual Preference Optimization module that introduces image-level preference pairs (diffusion-corrupted negatives) into the DPO framework.
  • Hierarchical Textual Preference Optimization combining response-, segment-, and token-level objectives in a single loss.
  • Empirical validation on LLaVA-1.6 (7B) and Muffin (13B) across four hallucination benchmarks, with public code and data.

Results

  • ObjHal (response-level R. / mention-level M., lower is better):
  • Muffin: 21.5/11.6 (base) → 13.1/6.6 (DPO) → 6.2/3.9 (CHiP); CHiP over DPO: 52.7% relative improvement
  • LLaVA-1.6: 14.1/7.4 (base) → 11.0/6.6 (DPO) → 4.9/3.2 (CHiP); CHiP over DPO: 55.5% relative improvement
  • CHiP vs. GPT-4V (13.6/7.3): LLaVA+CHiP beats GPT-4V by 64.0% / 56.2% relative on R./M.
  • AMBER (Hal / Cog, lower is better):
  • Muffin: Hal 32.1 → 26.5 → 17.6; Cog 3.5 → 2.5 → 1.5
  • LLaVA-1.6: Hal 48.6 → 38.9 → 24.5; Cog 4.2 → 3.0 → 1.6
  • MMHal Overall (higher is better): Muffin 2.4 → 2.5 → 2.6; LLaVA-1.6 2.8 → 2.7 → 2.9
  • HallusionBench (qA/fA, higher is better): LLaVA-1.6 15.8/20.8 → 22.2/28.3 → 23.5/26.0
  • Ablation (LLaVA, ObjHal R.): DPO 11.03 → HDPO alone 9.19 → CMDPO alone 9.76 → CHiP 4.92
  • Human evaluation (MMHal): CHiP outperforms DPO in 31.6% of distinguishable cases; ties in 63.5%.

Limitations

  • Rejected images are constructed by forward diffusion corruption (Gaussian noise at T=500), which may not capture semantically meaningful or ecologically valid visual negatives.
  • Visual encoder freezing during CHiP training yields slightly worse performance than keeping it trainable (Table 3), yet the paper does not deeply analyze why or recommend a configuration.
  • Evaluation is restricted to hallucination benchmarks; general VQA or reasoning capability changes are not assessed.
  • Segment-level hallucination identification relies on string-matching between chosen and rejected responses in RLHF-V, which may miss paraphrase-level hallucinations.
  • Only evaluated on two base MLLMs (LLaVA-1.6 7B, Muffin 13B); generalizability to larger or architecturally distinct models (e.g., InstructBLIP, Qwen-VL) is not demonstrated directly.

Relevance to Vision-Language Models

CHiP directly addresses a fundamental alignment failure in multimodal LLMs: standard DPO, despite using image-conditioned data, does not reliably close the gap between image and text embeddings or sharpen the model's ability to distinguish hallucinated content. By decomposing the DPO objective into visual and hierarchical textual components, CHiP provides a principled recipe for preference-tuning MLLMs with limited data (5k samples). This work fits into the growing literature on RLHF/DPO adaptations for vision-language alignment (alongside RLHF-V, FDPO, HALVA) and highlights that modality-specific preference signals—not just multimodal prompt conditioning—are necessary for effective hallucination reduction.