Skip to content

AVCD: Mitigating Hallucinations in Audio-Visual Large Language Models through Contrastive Decoding

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AVCD is a training-free contrastive decoding framework for audio-visual large language models (AV-LLMs) that dynamically identifies less dominant modalities via attention distributions and perturbs them with attentive masking to suppress cross-modal hallucinations. Unlike prior VLM contrastive decoding methods that corrupt a fixed modality, AVCD reformulates the CD objective for trimodal (audio, video, text) settings and adds entropy-guided adaptive decoding to reduce computational overhead.

Problem

Existing contrastive decoding (CD) methods for VLMs perturb a single fixed modality (vision or language), which is insufficient for AV-LLMs where hallucinations arise from complex unimodal and cross-modal interactions among audio, video, and language. No prior work had generalized CD to the trimodal setting, leaving a gap in inference-time hallucination mitigation for AV-LLMs.

Method

AVCD operates in three stages at inference time without any fine-tuning:

  1. Dominance-aware attentive masking: Computes a modality dominance score DM by summing attention weights of the final query token across all transformer layers, separately for audio, video, and text token spans. The modality with the highest average score is deemed dominant. For the non-dominant modalities, tokens exceeding the top-50% threshold of stacked attention values are zeroed out (attentive masking), generating perturbed logits biased toward the dominant modality.

  2. Trimodal contrastive formulation: Extends the standard two-modality CD formula by averaging original and corrupted distributions for each non-dominant modality (using a log-sum approximation via Taylor expansion), then summing per-modality CD terms. The final logit combines four forward-pass terms with separate contrastive coefficients αv and αa: logitAVCD = (2+αv+αa)logit(xv,xa,xl) + (1−αv+αa)logit(x¬v,xa,xl) + (1+αv−αa)logit(xv,x¬a,xl) − (αv+αa)logit(x¬v,x¬a,xl)

  3. Entropy-guided adaptive decoding (EAD): Skips the contrastive forward passes when the initial logit distribution has low entropy (high confidence, threshold τ=0.6), reducing the 4× forward-pass cost for already-certain tokens.

Key Contributions

  • First CD framework explicitly designed for trimodal AV-LLMs, handling audio–video–text interactions jointly.
  • Attention-based dominance scoring that dynamically selects which modalities to perturb rather than using a fixed perturbation strategy.
  • Attentive masking (zeroing top-attention tokens) instead of Gaussian noise injection, shown to produce smaller approximation errors than VCD.
  • Principled trimodal CD reformulation derived via logarithm approximation, with mathematical validation of approximation error on AVHBench.
  • Entropy-guided adaptive decoding that reduces inference overhead while preserving accuracy gains.
  • Demonstrated generalization beyond AV-LLMs to video-LLMs and image-LLMs.

Results

  • AVHBench (audio-visual hallucination QA):
  • VideoLLaMA2: Base 70.52% → AVCD 72.15% (+2.3%); VCD degrades to 65.18%
  • video-SALMONN: Base 58.19% → AVCD 62.18% (+7%); VCD improves slightly to 60.61%
  • MUSIC-AVQA (audio-visual QA):
  • VideoLLaMA2: Base 81.30% → AVCD 81.58%; VCD degrades to 77.66%
  • video-SALMONN: Base 48.50% → AVCD 49.73%; VCD degrades to 41.57%
  • AVHBench captioning score:
  • VideoLLaMA2: Base 2.84 → AVCD 3.03
  • video-SALMONN: Base 1.83 → AVCD 2.47
  • MSVD-QA (video-LLM):
  • VideoLLaMA2: Base 74.43% → AVCD 75.20%; Video-LLaVA: Base 70.20% → AVCD 72.16%
  • ActivityNet-QA (video-LLM):
  • VideoLLaMA2: Base 47.19% → AVCD 48.22%; Video-LLaVA: Base 47.48% → AVCD 48.03%
  • At EAD threshold Ï„=0.8: inference 2.25 s/token vs. VCD's 2.50, with accuracy 80.98% vs. Base 78.05% on AVHBench validation.
  • Approximation error (attentive masking vs. VCD noise) on AVHBench: Audio 0.032 vs. 0.083; AV 0.037 vs. 0.073.

Limitations

  • Requires multiple full attention maps at each decoding step; incompatible with FlashAttention, making it slower than base decoding even with EAD.
  • αv and αa hyperparameters require dataset-specific tuning (validated on 100 held-out samples per dataset).
  • EAD entropy threshold Ï„ is also a tunable hyperparameter with accuracy-speed sensitivity.
  • Tested on a small set of AV-LLM architectures (VideoLLaMA2, video-SALMONN); token-level fusion assumed—feature-level fusion models (video-SALMONN) require treating fused audio-visual as one modality.
  • Gains on video-LLMs and MUSIC-AVQA are modest (<1%), suggesting limited headroom in less hallucination-prone settings.

Relevance to Vision-Language Models

AVCD directly extends the contrastive decoding paradigm—well-established for VLMs (VCD, SID, CODE, ICD)—into the audio-visual multimodal regime, making it a natural next step for researchers tracking hallucination mitigation in VLMs. The attention-based dominance scoring and attentive masking strategies are modality-agnostic and could inform future CD methods in any multimodal LLM. The work also highlights that naive extension of VLM CD (e.g., applying VCD with an additional audio branch) actively degrades performance, underscoring that trimodal interactions require rethinking the contrastive objective rather than just adding a modality. For VLM researchers, this represents both a practical plug-and-play tool and a theoretical blueprint for scaling training-free hallucination suppression to higher modality counts.