Grounding Language with Vision: A Conditional Mutual Information Calibrated Decoding Strategy for Reducing Hallucinations in LVLMs

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CMI-VLD frames LVLM hallucination mitigation as a Conditional Pointwise Mutual Information (C-PMI) maximization problem and solves it via a bi-level optimization: a calibrated text token distribution that up-weights image-grounded tokens, and a lightweight learnable visual token purifier that discards image tokens irrelevant to the current response. The method is training-free for the base LVLM and introduces negligible inference overhead while consistently outperforming contrastive decoding baselines across five benchmarks.

Problem

LVLMs generate semantically plausible but visually unfaithful content because autoregressive decoding over-weights language priors and under-attends to visual tokens. Existing decoding-based mitigations manipulate text token distributions empirically, without a principled way to quantify or maximize the dynamic mutual dependency between the visual input and the ongoing generation; moreover, none jointly optimize both visual and textual token contributions to that dependency.

Method

Theoretical formulation. The hallucination objective is recast as maximizing C-PMI(V=v, Y=y | X=x), which decomposes step-wise as:

\[\sum_{t=0}^{l-1}\bigl[\log p_\theta(y_t \mid v, x, y_{<t}) - \log p_\theta(y_t \mid x, y_{<t})\bigr]\]

This induces a bi-level optimization with two alternating subproblems at each decoding step.

Inner subproblem — calibrated text sampling. At each step, the next-token distribution is replaced by:

\[p_c(\cdot \mid v, x, y_{<t}) = \text{softmax}\bigl[(1+\lambda)f_\theta(\cdot \mid v, x, y_{<t}) - \lambda f_\theta(\cdot \mid x, y_{<t})\bigr]\]

where λ controls calibration strength (optimal at 0.5). Adaptive token truncation prunes the candidate pool.

Outer subproblem — visual token purification. A lightweight purifier network P(·) (a few transformer blocks + MLP) takes concatenated image+text embeddings and outputs per-token retention probabilities. Binary masks are sampled via Gumbel-Softmax for differentiability. Selection also incorporates attention scores from LVLM layer 2 (empirically optimal), balancing C-PMI relevance and attention influence via hyperparameter α. A Frobenius-norm regularization enforces the target retention ratio γ. Training loss:

\[\mathcal{L} = [\log p_\theta(y_t|v,x,y_{<t}) - \log p_\theta(y_t|x,y_{<t})] + \alpha \cdot \text{Attn}_i(v) + \beta\|\text{sum}(M_v)/N - \gamma\|_F\]

The purified visual tokens v* feed into the calibrated text distribution, closing the bi-level loop.

Key Contributions

• Reformulates hallucination mitigation as C-PMI maximization, providing an information-theoretic foundation that unifies existing contrastive decoding methods as special cases (text-only C-PMI).
• Proposes the first bi-level decoding optimization jointly over visual and textual tokens.
• Designs a lightweight, trainable visual token purifier with Gumbel-Softmax enabling end-to-end gradient flow through discrete token selection.
• Achieves consistent SOTA across five benchmarks on multiple LVLM architectures with negligible inference overhead.

Results

• CHAIR (LLaVA-1.5, sampling): CS 30.2 / CI 9.3 vs. next-best VTI (CS 37.2 / CI 11.4) — 18.8% CS reduction.
• CHAIR (Shikra, greedy): CS 30.6 vs. VTI 36.8 (next best); CI 8.9 vs. VTI 11.7.
• GPT-4o SHR (LLaVA-1.5): 15.89% relative improvement in sentence-level hallucination ratio over competitive baselines.
• POPE (LLaVA-Next, greedy, adversarial): F1 82.14% vs. VASparse† 80.67% and SID 80.40%.
• MME (greedy): 1481.17 vs. next-best VCD 1472.57 and HALC† 1473.43.
• MMBench (greedy): 65.12 vs. OPERA† 64.78 (next best).
• Inference time: Comparable to or faster than SID and OPERA; removal of redundant visual tokens offsets purifier cost.

Limitations

• Requires training the visual token purifier on paired image-text data; generalization to unseen LVLM architectures requires retraining.
• Less effective on short yes/no probes (POPE) where only the first decoded token matters, since CMI-VLD is designed for full-sequence calibration.
• Introduces three sensitive hyperparameters (λ, α, γ) requiring per-model tuning.
• Experiments focus on LLaVA-1.5/NeXT, InstructBLIP, and Shikra; broader evaluation on larger-scale or closed-source LVLMs is absent.
• Reduction in response length (noted as a side effect) could affect downstream tasks requiring detailed descriptions.

Relevance to Vision-Language Models

This paper directly targets one of the most critical reliability failure modes of VLMs — hallucination — and provides a theoretically grounded decoding-time solution that requires no retraining of the base model. The C-PMI framework explicitly reveals why prior contrastive decoding methods work and where they fall short (text-only optimization), which has broad implications for the design of grounded multimodal generation. The visual token purification component also connects to the active line of work on efficient VLM inference via token sparsification, showing that reliability and efficiency can be co-optimized. Practitioners tracking VLM faithfulness, efficient inference, and decoding algorithms will find both the theoretical formulation and the empirical results directly applicable.