Breaking the Illusion: When Positive Meets Negative in Multimodal Decoding

🕒 Published (v1): 2026-04-22 15:11 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

Training-free positive-negative decoding reduces object hallucination; CVPR 2026

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

PND (Positive-and-Negative Decoding) is a training-free inference framework that combats object hallucination in VLMs by running three parallel forward passes—original, evidence-amplified, and evidence-suppressed—and combining their logits to enforce visual fidelity. It addresses both false-positive hallucination (invented objects) and false-negative hallucination (omitted objects), which single-path perturbation methods like VCD fail to handle simultaneously. PND achieves state-of-the-art results on POPE, MME, and CHAIR across LLaVA, InstructBLIP, InternVL, and Qwen-VL without any model retraining.

Problem

VLMs exhibit a Bayesian imbalance: deep-layer cross-modal attention progressively under-weights visual evidence (declining from ~13.7% in early layers to ~4.9% in late layers) while language priors dominate. Existing inference-time methods (VCD, VAF, AGLA) use only a single destructive perturbation path, which suppresses visual likelihood further and thus worsens false-negative hallucination (object omission) while partially addressing false-positive hallucination (object confabulation).

Method

PND constructs two complementary visual representations from the original image and uses them alongside the original in a three-path decoding step.

Salience estimation. Cross-modal attention maps \(\mathbf{A}_i\) are extracted from an external BLIP-ITM model across \(L\) layers. A fused salience map averages normalized per-layer maps: $\(\mathbf{M}_\text{fused} = \frac{1}{L}\sum_{i=1}^{L}\hat{\mathbf{A}}_i\)$

Positive pathway (\(V_\text{pos}\), amplifies visual likelihood): evidence-bearing patches are multiplicatively boosted: $\(\mathbf{V}_\text{pos} = \mathbf{V}_\text{orig} \odot (1 + \lambda \cdot \mathbf{M}_\text{fused})\)$

Negative pathway (\(V_\text{neg}\), isolates language prior): a consensus mask identifies minimal evidence regions (pixel-wise minimum across layers), and those regions are degraded using the DDPM forward noising process at noise level \(T\): $\(\mathbf{V}_\text{noise} = \sqrt{\bar{\alpha}_T}\,\mathbf{V}_\text{orig} + \sqrt{1-\bar{\alpha}_T}\,\boldsymbol{\epsilon}, \quad \boldsymbol{\epsilon}\sim\mathcal{N}(0,\mathbf{I})\)$ $\(\mathbf{V}_\text{neg} = \mathbf{V}_\text{orig}\odot(1-\mathbf{M}_\text{mask}) + \mathbf{V}_\text{noise}\odot\mathbf{M}_\text{mask}\)$

Decoding objective. Logits from all three paths are combined contrastively: $\(\mathbf{l}_\text{PND} = \mathbf{l}_\text{orig} + \alpha\,\mathbf{l}_\text{pos} - \gamma\,\mathbf{l}_\text{neg}\)$ A confidence threshold \(\beta\) masks out tokens implausible under the original distribution before the final softmax.

Key Contributions

• Empirical demonstration that cross-modal attention weight on visual patches decays from ~13.7% to ~4.9% across decoder layers, providing direct evidence for Bayesian imbalance.
• PND: a plug-and-play, training-free dual-path decoding framework addressing both false-positive and false-negative hallucination simultaneously.
• Use of BLIP-ITM as an architecture-agnostic differentiable proxy for disentangling evidence-bearing vs. prior-dominant visual regions.
• DDPM-based targeted feature degradation for the negative path—more semantically coherent than Gaussian noise, preventing OOD artifacts the model would ignore.
• New GPT-4.1-judged benchmark GCCCE evaluating Relevancy, Accuracy, Common Sense, and Fine-grained Precision for open-ended generation.

Results

POPE (object hallucination, Yes/No): - LLaVA-1.5-7B adversarial: 84.03% accuracy vs. 83.13% (AGLA), 81.13% (VCD), 78.53% (regular); F1 83.48 vs. 82.21 (AGLA) - InstructBLIP-7B adversarial: 82.20% accuracy vs. 81.13% (AGLA), 78.43% (VCD), 75.66% (regular) - Average improvement over greedy baseline: +6.4% accuracy, +5.5% F1

MME (perceptual capabilities): - LLaVA-1.5-7B total score: 621.67 vs. 608.67 (AGLA), 583.67 (VAF), 565.33 (VCD), 531.67 (regular) - InstructBLIP-7B total score: 423.33 vs. 406.67 (AGLA), 387.67 (VCD), 386.66 (regular) - Gains across both object-level (Existence, Count) and attribute-level (Position, Color) subtasks

CHAIR (open-ended captioning hallucination): - LLaVA-1.5-7B: \(\text{CHAIR}_s\) 46 vs. 47 (AGLA), 51 (VCD/regular); \(\text{CHAIR}_i\) 14.0 vs. 14.2 (AGLA), 16.7 (VCD) - InstructBLIP-7B: \(\text{CHAIR}_s\) 42 vs. 46 (AGLA), 59 (VCD), 58 (regular); \(\text{CHAIR}_i\) 11.2 vs. 12.3 (AGLA)

Generalization: consistent gains across LLaVA-1.5-13B, InstructBLIP-13B, Qwen-VL-7B, Qwen3VL-2B, and InternVL2-2B.

Limitations

• Adds inference overhead from three parallel forward passes plus external BLIP-ITM attention extraction (overhead not quantified in the excerpted text).
• The positive pathway depends on BLIP-ITM attention quality; misaligned attention maps from BLIP could yield incorrect salience estimates.
• Hyperparameters \(\alpha\), \(\gamma\), \(\lambda\), \(\beta\), \(\tau\), and DDPM noise level \(T\) require tuning per model or task.
• GCCCE benchmark relies on GPT-4.1 as judge, introducing proprietary model dependency and potential judge bias.
• Evaluation restricted to open-source models; behavior on proprietary VLMs (GPT-4V, Gemini) is not assessed.

Relevance to Vision-Language Models

PND directly addresses one of the most persistent failure modes in current VLMs—object hallucination driven by over-reliance on language priors—without modifying model weights, making it immediately applicable to any deployed VLM. The dual-path contrastive decoding approach is a principled extension of the VCD line of work, adding a complementary positive pathway that VCD-style single-path methods structurally cannot provide, and is therefore relevant to any researcher studying inference-time alignment or decoding strategies. The empirical finding that visual attention weight falls from ~13.7% to ~4.9% across decoder depth is a concrete diagnostic that informs both architectural design (e.g., attention rebalancing) and training objectives (e.g., visual grounding losses). The method's demonstrated generalization across diverse fusion architectures (projection-based LLaVA, query-transformer InstructBLIP, InternVL, Qwen-VL) positions it as a broadly applicable component in the VLM toolkit.