Skip to content

Learning from Fine-Grained Visual Discrepancies: Mitigating Multimodal Hallucinations via In-Context Visual Contrastive Optimization

🕒 Published (v1): 2026-05-29 13:44 UTC · Source: Arxiv · Venue: ICML 2026 · link

Why this paper was selected

ICML 2026; visual contrastive DPO for multimodal hallucination mitigation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Multimodal hallucination in VLMs is exacerbated by two flaws in visual preference DPO: a partition-function mismatch that makes the optimization objective theoretically inconsistent, and coarse-grained negative images that enable shortcut learning. IC-VCO fixes both by placing both images in a shared multi-image context (eliminating the partition function asymmetry) and generating hard negatives via surgical image editing rather than retrieval or synthesis.

Problem

Standard DPO treats images as static context, providing no explicit visual grounding signal. Existing visual preference DPO methods contrast responses conditioned on different images (\(m\) vs. \(m'\)), but this violates DPO's theoretical assumptions: the partition functions \(Z(m,x)\) and \(Z(m',x)\) differ and do not cancel, leaving an intractable residual bias \(\log\frac{Z(m,x)}{Z(m',x)}\) that arbitrarily shifts the implicit decision boundary per sample. Additionally, negatives constructed via text-to-image synthesis or retrieval exhibit global stylistic drift, allowing models to reject \(m'\) on low-level visual cues rather than fine-grained semantic differences.

Method

IC-VCO restructures visual preference optimization around a shared multi-image context \(M = [m, m']\):

  1. In-Context Visual Contrastive Optimization: Both original and contrastive images are concatenated into a single context \(M\). An anchor prompt extension (\(\hat{x}\), \(\hat{x}'\)) explicitly instructs the model which image to ground its response on. Since chosen and rejected responses share the identical condition \((M, \hat{x})\), the partition function cancels exactly, recovering a theory-consistent DPO objective (Eq. 9).

  2. Visual Contrast Distillation (VCDist): A train-inference gap exists because IC-VCO trains on multi-image contexts but inference uses a single image. VCDist uses the multi-image win probability \(p_\text{multi}\) as a soft teacher to regularize the single-image branch \(p_\text{single}\). A dual gate enforces distillation only when the teacher is correct (\(p_\text{multi} > 0.5\)) and more confident than the student (\(p_\text{single} < \text{sg}(p_\text{multi})\)), with stop-gradient applied to teacher logits (Eq. 11).

  3. Contrastive Sample Editing: Negatives are generated via targeted local edits rather than global synthesis. A QwenVL-Plus planner produces an edit instruction \(T\) targeting three hallucination types (existence, attribute, relation) under two scenarios (hallucination realization and injection). Qwen-Image-Edit applies \(T\) with reversible padding, and a verification step discards failed edits (91% success rate over 19,453 samples). CLIP similarity between edited pairs (mean 94.72) substantially exceeds synthesis-based pairs (mean 72.88).

  4. Token-level Fine-grained Preference: For single-image DPO loss, a token mask restricts preference scoring to tokens describing the edited visual evidence, increasing sensitivity to fine-grained discrepancies.

The full training objective \(L_\text{Total} = L_\text{IC-VCO} + L'_\text{IC-VCO}\) combines symmetrical multi-image and single-image DPO losses with anchor regularizers and the VCDist term (Eq. 14–15).

Key Contributions

  • Theoretical identification of the partition function mismatch in visual preference DPO and a mathematically rigorous fix via shared multi-image context.
  • VCDist: a dual-gated distillation regularizer that bridges the multi-image training / single-image inference gap.
  • Contrastive Sample Editing pipeline producing fine-grained hard negatives via localized image editing, shown to improve multiple baselines beyond IC-VCO itself.
  • Token-level preference scoring aligned with edited semantic regions.

Results

  • On LLaVA-NeXT-Interleave-Qwen-7B, IC-VCO with synthetic samples achieves overall score 62.83, exceeding the strongest baseline (SymMPO, 61.50) by +1.33; with edited samples, 63.35 vs. SymMPO 62.11 (+1.24).
  • On LLaVA-OneVision-Qwen2-7B, IC-VCO surpasses the strongest baseline by +0.26 (synthetic) and +0.48 (edited).
  • Replacing synthetic negatives with edited negatives improves all baselines on both backbones (NeXT-Interleave baselines: +0.08–+0.61; OneVision baselines: +0.25–+1.23), validating the generality of the editing strategy.
  • IC-VCO shows strongest gains on attribute- and existence-oriented benchmarks (AMBER-Attr, AMBER-Exist, CRPE-Exist) and BLINK visual perception; relation-centric metrics (AMBER-Rel, CRPE-Rel) are more mixed, attributed to low relation-edit coverage (828/19,453 samples).
  • Evaluated across five benchmarks: HallusionBench, AMBER, CRPE, R-Bench, BLINK.

Limitations

  • Relation-type edits are underrepresented (828 of 19,453 samples), leaving relation hallucination less well addressed.
  • The editing pipeline depends on QwenVL-Plus and Qwen-Image-Edit; edit success rate is 91%, meaning ~9% of candidate samples are discarded.
  • VCDist introduces a train-inference distributional gap (multi-image training vs. single-image inference) that is mitigated but not eliminated.
  • Evaluated only on 7B-scale LLaVA variants; generalization to other architectures or scales is not demonstrated.
  • The symmetric DPO objective still relies on a single-image DPO branch whose theoretical status under visual preference is not fully resolved.

Relevance to Vision-Language Models

IC-VCO directly addresses a known weak link in VLM post-training: the failure of standard RLHF/DPO pipelines to enforce visual grounding, which underlies the widespread "visual neglect" phenomenon. The theoretical critique of partition-function mismatch is broadly applicable to any visual preference DPO variant and constitutes a meaningful methodological contribution to the multimodal alignment literature. The contrastive editing pipeline is orthogonal to the optimization objective and can be dropped into existing methods (mDPO, S-VCO, SymMPO), making it a reusable infrastructure component. For researchers tracking VLMs, this work represents a consolidation of fine-grained visual supervision, DPO theory, and targeted data curation into a unified framework for hallucination mitigation.