Skip to content

Systematic Reward Gap Optimization for Mitigating VLM Hallucinations

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TPR (Topic-level Preference Rewriting) is a DPO data curation framework that mitigates visual hallucinations in VLMs by decomposing responses into semantic topics, self-resampling alternatives within each topic, and selectively replacing topic units to precisely engineer reward gaps in preference pairs. A curriculum learning variant (TPR-CL) progressively introduces harder rejected responses, achieving state-of-the-art hallucination reduction. TPR-CL reduces hallucinations by ~93% on Object-HalBench and ~41% on MMHal-Bench over the LLaVA-1.5-7B base.

Problem

Existing DPO-based hallucination mitigation methods curate preference pairs via ranking (selecting from potentially flawed outputs, yielding weak reward gaps) or external rewriting (using black-box models like GPT-4V, which introduce hallucination types misaligned with the target model's intrinsic failure modes). Neither approach enables systematic, fine-grained control over the reward gap configuration — the deliberate shaping of how discriminative each (y_w, y_l) pair is — leading to suboptimal alignment signal and poor data efficiency.

Method

TPR operates in three stages on a reference model (LLaVA-1.5-7B) with a labeler model (LLaVA-NeXT-34B):

  1. Topic-level alternatives generation: Sample M candidate responses, decompose each into fine-grained semantic units via the reference model, then cluster units across responses by textual consistency (LLM query: "do these describe the same topic?") and visual correlation (CLIP features). Within each topic cluster, perform intra-topic self-resampling: convert units to wh-questions and re-query the reference model to generate diverse alternative phrasings, avoiding external bias.

  2. Selective topic replacement: Score each alternative via yes/no questions to the labeler model (score = p_Yes − p_No). Construct y_w and y_l by selecting a random response as template, then replacing each topic unit with the highest- and lowest-scoring alternatives respectively. Apply in-context rewriting via the reference model to maintain fluency.

  3. Curriculum learning (TPR-CL): Two-stage DPO training. Warm-up (60% of data): greedy replacement maximizing score divergence. Hard-mining (40%): progressively raise the score of the rejected alternative toward the decision boundary, forcing the model to distinguish subtle hallucinations.

Key Contributions

  • Identifies reward gap configuration as the key underexplored axis in DPO-based hallucination mitigation.
  • TPR framework providing topic-level, self-contained (no external model) fine-grained control over preference pair semantics.
  • Intra-topic self-resampling that better matches the target model's intrinsic hallucination distribution compared to external rewriters.
  • Curriculum learning strategy (TPR-CL) that progressively optimizes the global reward gap configuration via hard negative mining at the topic level.
  • Superior data efficiency: TPR with 20k samples surpasses human-annotated RLHF-V (1.4k) and RLAIF-V (requiring effectively 88k samples due to multi-round retraining).

Results

  • Object-HalBench: TPR-CL achieves CHs=3.4, CHi=1.8, vs. base LLaVA-1.5-7B at CHs=53.6, CHi=25.2 (~93% reduction); vs. prior SOTA HSA-DPO-13B at CHs=5.3, CHi=3.2.
  • MMHal-Bench: Score 3.06 (TPR-CL) vs. 2.36 (base); Hall. 30.2 vs. 51.0 (~41% reduction).
  • AMBER: Acc=82.7, F1=87.8 (TPR-CL) vs. 73.5/77.6 (base); competitive with best prior methods.
  • POPE: Adv.=84.2, All=87.6 (TPR-CL).
  • RefoMB Trust.=61.1, Win.=32.3 (TPR-CL) vs. 30.8/12.1 (base), outperforming all automated methods.
  • General benchmarks (LLaVA-Bench, MMStar): maintained or improved vs. base (e.g., LLaVA-Bench Acc: 71.1 vs. 59.7).
  • Ablations confirm: random replacement (2b) degrades CHs from 5.9→29.7; omitting in-context rewriting (2d) degrades CHs to 35.5; ranking-based baseline (2g) yields CHs=9.7; external rewriting (2i,2j) yields CHs=11.3–15.6.
  • Outperforms previous methods by ~20% average across hallucination benchmarks.

Limitations

  • Multi-step pipeline (decomposition → clustering → resampling → ranking → selective replacement → in-context rewriting) increases computational overhead.
  • Clustering step may be redundant and is flagged by authors for future simplification.
  • Curriculum learning strategy explores only one dimension of reward gap optimization; richer curation dimensions remain uninvestigated.
  • Currently addresses perceptual hallucinations only; extension to multimodal logical/causal reasoning hallucinations is left as future work.
  • Evaluation limited to LLaVA-1.5-7B as the policy model; generalization to other VLM architectures is partially explored only in appendix.

Relevance to Vision-Language Models

TPR directly advances the DPO alignment methodology for VLMs by providing a principled, model-intrinsic mechanism to engineer preference data quality — a critical bottleneck in current RLHF/DPO pipelines for multimodal models. The insight that hallucination type distribution of rejected responses must match the target model's intrinsic failure modes (rather than patterns introduced by an external rewriter) is practically important for anyone building alignment pipelines for VLMs. The curriculum learning strategy also connects to broader questions about training dynamics and difficulty scheduling in preference learning, relevant to scaling robust multimodal alignment. The demonstrated data efficiency advantage (20k TPR vs. 88k effective RLAIF-V) has direct implications for cost-effective VLM fine-tuning at scale.