Skip to content

Enhancing Vision-Language Model Reliability with Uncertainty-Guided Dropout Decoding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

DROPOUT DECODING is an inference-time method that reduces object hallucinations in large vision-language models (LVLMs) by quantifying epistemic uncertainty of visual tokens via text-space projection, then selectively masking high-uncertainty tokens across an ensemble of forward passes. It requires no additional training or model modification and consistently outperforms contrastive/attention-based decoding baselines on CHAIR, THRONE, and MMBench.

Problem

LVLMs misinterpret or over-attend to ambiguous/noisy visual tokens during autoregressive decoding, producing hallucinated objects. Existing remedies either require expensive fine-tuning or rely on heuristic attention/logit adjustments with no principled uncertainty grounding. Standard dropout cannot be applied to frozen pretrained parameters at inference time.

Method

Text-space projection of visual tokens. Each visual token \(x^v_i\) is forward-passed through the LLM decoder alone (using the logit lens principle), producing a visual-textual distribution \(q^{\text{proj}}_i = \text{softmax}(W_V h^v_i)\) over the text vocabulary. The average across all visual tokens gives a "baseline" distribution \(\bar{q}^{\text{proj}}\).

Uncertainty decomposition. Total perception uncertainty \(H(\bar{q}^{\text{proj}})\) decomposes into aleatoric (\(H(q^{\text{proj}}_i)\), entropy of individual token projection) and epistemic (\(D_{\text{KL}}(q^{\text{proj}}_i \| \bar{q}^{\text{proj}})\), divergence from the mean). Epistemic uncertainty is used because it correlates with informative but potentially misinterpreted patches.

Uncertainty-guided token dropout. At each decoding step, \(K\) dropout masks are sampled; the probability of dropping token \(i\) scales with its normalized epistemic uncertainty (Eq. 9, controlled by range parameters \(\gamma^{(k)}\) and \(\delta^{(k)}\)). An optional preliminary forward pass identifies "relevant" tokens for the current generation step (those whose preliminary prediction appears in the token's top-\(k\) projected vocabulary), which are then pinned as retained. \(K\) masked forward passes are batched and their output tokens aggregated via majority voting; ties go to the least-masked candidate.

Key Contributions

  • Supervision-free, training-free uncertainty quantification at the visual token level by repurposing the LLM decoder's own projections (logit lens).
  • Novel decomposition of per-token perception uncertainty into aleatoric and epistemic components without model ensembles.
  • Uncertainty-guided input-token dropout at inference time, drawing the analogy to parameter-space dropout but applied to context tokens.
  • Ensemble decoding with majority voting over masked contexts for robust, hallucination-reduced generation.

Results

  • CHAIR (LLaVA-1.5): CHAIRS 39.80 vs. 42.20 (greedy) and 41.47 (OPERA); CHAIRI 11.73 vs. 12.83 (greedy).
  • CHAIR (InstructBLIP): CHAIRI 6.63 vs. 7.90 (greedy), 6.93 (beam search), 8.23 (OPERA); CHAIRS 24.53 vs. 25.87 (beam), 27.87 (greedy) — 16%/12% improvement over second-best.
  • CHAIR (LLaVA-NEXT): CHAIRS 26.26 vs. 28.80 (greedy); CHAIRI 7.39 vs. 8.10 (greedy).
  • THRONE (LLaVA-1.5): \(P_{\text{all}}\) 0.772 vs. 0.759 (greedy); \(F^{0.5}_{\text{all}}\) 0.796 vs. 0.784 (greedy).
  • MMBench: LLaVA-1.5: 74.01 vs. 73.86 (OPERA, second-best); LLaVA-NEXT: 74.31 vs. 74.54 (OPERA, best baseline; DROPOUT DECODING 74.31 — slight under-performance here).
  • Efficiency (w/o preliminary pass): only ~7.8% throughput reduction vs. greedy; GPU memory increase negligible (14.02 → 15.31 GB on HuggingFace Transformers).
  • Optimal ensemble size is \(K=3\); random masking (ablation) causes repetitive generation and fails THRONE, validating the epistemic uncertainty criterion.

Limitations

  • The preliminary forward pass (full optional step) roughly doubles per-token inference cost (45.7% throughput reduction, 84.4% wall-time increase).
  • Performance gain on LLaVA-NEXT MMBench (74.31) is slightly below OPERA (74.54), suggesting the approach is not universally best across all metrics.
  • Hyperparameters (\(\gamma^{(k)}\), \(\delta^{(k)}\), \(K\)) require tuning; aggressive dropout (\(\gamma^{(4)}=0.9\)) degrades InstructBLIP.
  • Logit-lens projection is a heuristic; visual tokens at positions before the text query are not explicitly trained to produce text distributions, so projection quality may vary by architecture.
  • Experiments limited to three LVLMs (LLaVA-1.5, InstructBLIP, LLaVA-NEXT); generalization to newer architectures (e.g., those with interleaved visual tokens) is unverified.

Relevance to Vision-Language Models

DROPOUT DECODING directly targets the object hallucination problem endemic to LVLMs, offering a plug-in inference-time fix compatible with any frozen pretrained model — highly relevant to practitioners deploying LVLMs where fine-tuning is infeasible. The method's use of the LLM decoder's own vocabulary projections as a proxy for token informativeness connects to the mechanistic interpretability literature (logit lens) and extends it to VLM uncertainty quantification. It complements contrastive decoding methods (VCD) and attention-penalty methods (OPERA) by addressing the problem at the level of input context rather than output logits. For researchers tracking VLMs, this work establishes epistemic uncertainty of visual tokens as a principled and empirically superior selection criterion compared to attention-based heuristics.