Skip to content

Importance Sampling for Multi-Negative Multimodal Direct Preference Optimization

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; importance-sampling multi-negative multimodal DPO for VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MISP-DPO extends multimodal Direct Preference Optimization by replacing single-negative image comparisons with semantically diverse multi-negative sets, selected via sparse autoencoder (SAE) decomposition in CLIP space. A Plackett–Luce ranking objective and an importance-sampling gradient estimator unify these negatives into an efficient, theoretically grounded training signal. Across five benchmarks and three VLM backbones, MISP-DPO consistently outperforms all single-negative baselines.

Problem

Existing multimodal DPO methods generate only one negative image per training pair (via cropping, diffusion perturbation, or similarity retrieval), collapsing visual supervision to a single deviation axis. This causes the model to learn spurious shortcuts (e.g., "avoid green hues" instead of "avoid object mismatches") and leaves orthogonal failure modes—object identity, color, spatial layout, contextual mismatch—entangled and unexplored, sustaining hallucinations and limiting generalization.

Method

MISP-DPO operates in two stages:

Stage 1 — Diverse negative selection. Positive image \(m_p\) and prompt \(x\) are encoded by CLIP encoders \(f_v, f_t\) and fused via outer product: \(e = \mathrm{vec}(h_v \times h_t^\top) \in \mathbb{R}^{d^2}\). For each candidate negative \(m_n^i\) a difference vector \(d_i = e(m_p, x) - e(m_n^i, x)\) is formed. A sparse autoencoder (SAE) with reconstruction + KL-sparsity loss decomposes \(d_i\) into disentangled latent factors. Each candidate is scored by $\(s_i = \frac{\|E(d_i)\|_1}{\max_j \ell_j} + \frac{\|d_i - D(E(d_i))\|_2^2}{\max_j v_j}\)$ prioritizing high reconstruction error (informativeness) and large activation magnitude. A greedy diversity-promoting selection (Algorithm 1) adds to \(\tilde{S}_n\) the candidate maximizing \(s_i + \beta \min_{j \in \tilde{S}_n}(1 - \cos(E(d_i), E(d_j)))\), yielding \(K=3\) negatives that cover distinct semantic directions.

Stage 2 — Plackett–Luce multi-negative objective with importance sampling. The visual preference loss generalizes multimodal DPO to \(N\) negatives: $\(L_{\mathrm{img}}(\theta; \tilde{S}_n) = \log\sigma\!\left(-\log\sum_{i\in\tilde{S}_n}\exp\!\left(\beta\log\frac{\pi_\theta(y_p|x,m_n^i)}{\pi_{\mathrm{ref}}(y_p|x,m_n^i)} - \beta\log\frac{\pi_\theta(y_p|x,m_p)}{\pi_{\mathrm{ref}}(y_p|x,m_p)}\right)\right)\)$ The gradient decomposes as a distribution \(p_\theta(m_n^i|x,m_p,y_p)\)-weighted sum over per-negative correction signals (Lemma 4.1). Because sampling from \(p_\theta\) is intractable, the learnable proposal \(q_\phi\) is used; importance weights \(p_\theta/q_\phi\) correct the bias, yielding an estimator with MSE \(O(1/K)\). A text DPO loss \(L_{\mathrm{text}}\) is added, and the final loss is \(L = L_{\mathrm{img}} + \lambda L_{\mathrm{text}}\) with \(\lambda=1\).

Key Contributions

  • First multimodal DPO framework using semantically diverse multiple negatives rather than single-axis perturbations.
  • SAE-guided importance sampling in CLIP space that disentangles latent visual error factors into interpretable, orthogonal directions.
  • Plackett–Luce multi-negative objective with a provably unbiased importance-sampling gradient estimator (MSE \(O(1/K)\)).
  • Greedy diversity-promoting selection algorithm ensuring coverage of heterogeneous failure modes.
  • Demonstrated consistent gains across three VLM backbones (LLaVA-1.5-7B, Qwen2.5-VL-7B, Qwen2.5-VL-3B) on five benchmarks.

Results

  • LLaVA-1.5-7B: +30.09% average improvement over BASE; MMHalBench hallucination rate 32.29% vs. 51.04% (BASE); MMVP accuracy 63.00 vs. 60.67 (BASE); outperforms mDPO and CHiP on all metrics.
  • Qwen2.5-VL-7B: +5.35% over BASE; MMHalBench HalRate 11.46% (tied with DPO, best among multimodal methods); POPE Acc. 88.66 vs. 87.65 (BASE); MMVP 79.00 vs. 77.67.
  • Qwen2.5-VL-3B: +19.89% over BASE; MMHalBench HalRate 13.54% vs. 22.34%; WildVision WinRate 52.4 vs. 46.6 (BASE), best of all methods; MMVP 74.25 vs. 70.60.
  • Negative sampling ablation (Table 2): MISP-DPO outperforms diffusion, crop+diffusion, and similarity-retrieval variants on nearly every metric for both Qwen2.5-VL-7B and 3B.
  • Random multi-negative sampling improves over single-negative baselines but consistently underperforms SAE-guided selection.
  • Optimal \(K=3\) negatives per instance; performance degrades beyond \(K=4\) on most benchmarks (Figure 2 right).

Limitations

  • Negative pool is sourced from COCO training images; out-of-distribution or domain-specific tasks may require different pools.
  • SAE training adds a preprocessing overhead not counted in per-training-step cost comparisons.
  • Importance-sampling variance bound depends on max importance weight, which can be large if \(q_\phi\) and \(p_\theta\) diverge significantly.
  • Evaluated only on 7B-class models; scaling behavior to larger VLMs is unverified.
  • Text negative construction follows prior work without novel contributions; improvements are primarily on the visual side.

Relevance to Vision-Language Models

MISP-DPO directly addresses a core alignment failure in VLMs: hallucinations that persist because preference training under-specifies the space of visual errors. By decomposing visual deviations into disentangled latent factors via SAE and ranking across multiple diverse negatives through the Plackett–Luce model, the method provides richer, more structured supervision than any prior multimodal DPO approach. The use of CLIP as a shared embedding space for both selection and training connects naturally to the contrastive pre-training paradigm underlying most modern VLMs, making the framework broadly applicable. This work advances the line of VLM alignment research (mDPO, CHiP, S-VCO) by demonstrating that the quality and diversity of negative examples—not just the objective form—is a primary determinant of preference optimization effectiveness in multimodal settings.