Skip to content

P\(^2\)-DPO: Grounding Hallucination in Perceptual Processing via Calibration Direct Preference Optimization

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Calibration DPO grounds hallucination in perceptual processing; preference learning

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

P\(^2\)-DPO is a self-correcting DPO framework for Large Vision-Language Models (LVLMs) that eliminates hallucination arising from perceptual processing failures—distinct from perception failures—by constructing on-policy, vision-aware preference pairs without any human annotation. It introduces two contrastive pair types (Focus-and-Enhance and Visual Robustness) and a Calibration Loss that causally attributes correct answers to visual inputs, outperforming human-feedback baselines on standard hallucination benchmarks.

Problem

Existing DPO-based hallucination mitigation methods are (1) vision-agnostic: preference pairs are built from textual differences alone, providing poor gradient signal for vision-dominant parameters \(\theta_1\); and (2) off-policy: reliance on external human or AI feedback yields preferred responses with \(\pi_\text{ref}(y^w | I, P) \to 0\), causing the sigmoid weighting \(\sigma(\hat{r}_l - \hat{r}_w) \to 0\) and nullifying gradient updates. Both problems leave two perceptual processing failure modes unaddressed: (a) a Perceptual Bottleneck where the model's attention correctly localizes a region yet still hallucinates, and (b) Lack of Robustness where imperceptible image noise triggers hallucination.

Method

P\(^2\)-DPO generates all preference pairs autonomously from the reference model, operating on a single \((I, P)\) instance:

Focus-and-Enhance Pairs. A forward pass through \(M_\text{ref}\) with attention-guiding prompts yields the final answer-to-image attention map \(A = A_\text{tok} \otimes A_\text{img} \in \mathbb{R}^{L \times L_c \times 1 \times N^2}\). An adaptive cropping algorithm extracts the salient crop \(I_\text{crop}\). The enhanced input \(I_\text{aug} = \text{Combine}(I, I_\text{crop})\) (with a refined prompt) generates \(y^w_\text{focus}\); the degraded input \(I_\text{deg} = \text{Erase}(I, \text{Bbox}(I_\text{crop}))\) generates \(y^l_\text{focus}\).

Visual Robustness Pairs. Gaussian noise produces \(I_\text{noise}\); the losing response \(y^l_\text{rob} = M_\text{ref}(I_\text{noise}, P)\). A Contrastive Amplification decoding strategy then refines the winning response: $\(y_t \sim \text{softmax}\bigl((1+\lambda_\text{ca})\cdot\text{logits}_\text{EP}(y_t) - \lambda_\text{ca}\cdot\text{logits}_\text{AT}(y_t)\bigr), \quad y_t \in V_\text{head}\)$ where the expert (EP) conditions on \(I\) and the amateur (AT) on \(I_\text{noise}\).

Calibration Loss (\(L_\text{Calib}\)). Beyond the standard DPO loss on Focus-and-Enhance pairs, a complementary objective maximizes the Perceptual Confidence Gain \(\Delta_\pi(y) \triangleq \log\pi(y|I_\text{aug}) - \log\pi(y|I_\text{deg})\), equivalently maximizing the Visual Information Dependency \(I(Y^w_\text{focus}; F_v^+ | P, \theta)\): $\(L_\text{Calib} = -\mathbb{E}\left[\log\sigma\!\left(\beta\log\frac{\pi_\theta(y^w|I,I_\text{crop},P)}{\pi_\theta(y^w|I_\text{deg},P)} - \beta\log\frac{\pi_\text{ref}(y^w|I,I_\text{crop},P)}{\pi_\text{ref}(y^l|I_\text{deg},P)}\right)\right]\)$

Dynamic Deficit-Weighting (DDW). A CLIP-based Perceptual Gain Ratio \(r = \frac{\text{CLIPScore}(P, I_\text{crop})}{\text{CLIPScore}(P, I)}\) determines per-sample adjustment \(\alpha = \alpha_\text{max}\cdot\tanh\!\left(\frac{r-1.0}{\tau}\right)\), dynamically balancing \(w_\text{focus}\) and \(w_\text{robust}\) in the total loss \(L_\text{total} = \mathbb{E}[w_\text{focus}\cdot L_\text{focus} + w_\text{robust}\cdot L_\text{dpo\_rob}]\).

Pairs are filtered by perplexity threshold \(\tau_\text{ppl}\) and log-probability margin range \([\theta_\text{low}, \theta_\text{high}]\).

Key Contributions

  • Taxonomizes visual hallucination causes into Perception vs. Perceptual Processing failures and identifies Perceptual Bottleneck and Lack of Robustness as two underexplored manifestations of the latter.
  • Proposes Visually-Grounded Contrastive Preference Generation (VCPG): on-policy, vision-aware preference pairs constructed entirely from the model's own outputs, with information-theoretic analysis (Visual Information Dependency, Fisher Information) proving VCPG provides stronger gradient signal on \(\theta_1\) than PSC.
  • Introduces the Calibration Loss \(L_\text{Calib}\) that causally ties winning responses to visual interventions rather than learning a purely correlational signal.
  • Introduces Dynamic Deficit-Weighting to adaptively balance two orthogonal training signals per sample using pre-trained CLIP.
  • Demonstrates annotation-free training: preference data sourced from RLHF-V prompts but no human labels used.

Results

  • Perceptual Bottleneck (TextVQA/ARF metric): P\(^2\)-DPO achieves AFR = 18.71 vs. 15.57 (DPO\(_\text{RLHF}\)) and 14.73 (LLaVA-1.5-7B base); Processing Accuracy improves from 66.29% (base) to 70.10% (+3.81 pp) vs. 65.71% for DPO\(_\text{RLHF}\) (a regression of −0.58 pp).
  • Hallucination benchmarks (POPE, HallusionBench, MMHal-Bench, AMBER): P\(^2\)-DPO outperforms "strong baselines that rely on costly human feedback" with comparable training data; exact numbers for all benchmarks are in the paper but not fully reproduced in the excerpt.
  • Generalizability demonstrated on LLaVA-1.5-7B, Qwen2.5-VL-7B, and Qwen2.5-VL-3B.

Limitations

  • Evaluated primarily on LLaVA-1.5-7B; stronger proprietary models are not tested.
  • Attention-map-based cropping requires a full forward pass per sample, adding inference overhead at data-generation time.
  • The Contrastive Amplification hyperparameter \(\lambda_\text{ca}\) and DDW parameters \(\alpha_\text{max}, \tau\) require tuning.
  • Data generation bootstraps from RLHF-V prompts; quality dependence on the seed dataset is not fully analyzed.
  • Robustness evaluation is limited to Gaussian noise; other corruption types are not assessed.

Relevance to Vision-Language Models

This paper directly addresses the DPO alignment pipeline for LVLMs, identifying a fundamental flaw in how preference data is constructed for multimodal models—the vision-agnostic, off-policy nature of existing datasets. The theoretical grounding via Visual Information Dependency and Fisher Information provides a principled framework for understanding why visual preference pairs should differ at the image-input level rather than the text-output level, which has broad implications for any LVLM alignment method. The self-correcting, annotation-free paradigm is practically significant as it scales without human labeling costs. For VLM researchers, the Calibration Loss formulation and the Perceptual Confidence Gain concept offer reusable components for grounding model outputs in visual evidence.