Skip to content

The Sound of Absence: Audio-Language Embedding Models Struggle with Negation

🕒 Published (v1): 2026-07-14 02:55 UTC · Source: Arxiv · link

Why this paper was selected

Hung-yi Lee (NTU); exposes fundamental negation blindspot in CLAP-style audio-language models

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Audio-language embedding models (CLAP, M2D-CLAP, WAVE) systematically fail to distinguish negated from affirmative sound descriptions, mapping "cat meowing" and "no cat meowing" to nearly identical embeddings. The authors introduce NegEval-Audio, a framework that converts existing audio-caption datasets into two negation-aware tasks, exposing this affirmation bias as a fundamental geometric flaw rather than a surface artifact. A training-free embedding steering method partially mitigates the problem for multiple-choice tasks but yields only marginal retrieval gains.

Problem

Existing audio-language retrieval benchmarks evaluate only affirmation-only semantics—whether a model recognizes sounds that are present—never whether it correctly excludes sounds that a query explicitly negates (e.g., "rain without thunder"). This evaluation blind spot hides a systematic affirmation bias in the joint embedding space, where negated and affirmative captions occupy nearly identical regions.

Method

NegEval-Audio transforms existing audio-caption datasets (AudioCaps, Clotho) without new audio collection:

  1. Positive concept extraction: An LLM parses each caption \(c\) into \(P(a) = \{p_1, \ldots, p_m\}\)—the set of present sound events.
  2. Negative concept proposal and verification: An LLM proposes contextually plausible absent events \(\tilde{N}(a)\); an audio-aware LLM (Qwen2.5-Omni-7B) verifies each candidate for true absence, yielding \(N_v(a)\) (96.7% human agreement on 460 samples).
  3. Task construction:
  4. Retrieval-Neg: Original caption appended with "There is no sound of {absent sound}", paraphrased for fluency; standard recall@K evaluation.
  5. MCQ-Neg: Four-option questions with options differing only in affirmation/negation of specific concepts, categorized as Affirmation, Negation, or Hybrid types.
  6. Training-free steering: Given a negated caption embedding \(e_C\) and negated concept embedding \(e_{C_{\text{neg}}}\), the corrected embedding is \(e^* = e_C - \lambda e_{C_{\text{neg}}}\) with \(\lambda = 0.2\) fixed across all settings.

Key Contributions

  • Identifies negation handling as a critical, previously overlooked failure mode in audio-language embedding models.
  • Introduces NegEval-Audio, a dataset-agnostic pipeline converting affirmation-only benchmarks into negation-aware retrieval and MCQ tasks.
  • Demonstrates that CLAP, M2D-CLAP, and MLLM-based WAVE all exhibit severe affirmation bias (Negation-type MCQ accuracy 0.2–7.1% vs. 25% random chance).
  • Provides controlled text-side diagnosis showing negated descriptions cluster near both their affirmative counterparts and bag-of-concepts representations.
  • Shows training-free steering improves MCQ-Neg locally (WAVE-7B: +11.0 pp on AudioCaps) but fails to reshape global retrieval geometry.

Results

  • Retrieval-Neg vs. Standard Retrieval (AudioCaps): Average R@1 drops from 34.1 → 26.1; R@5 from 69.5 → 59.8; R@10 from 82.8 → 75.1 across all eight checkpoints.
  • M2D-CLAP-2025 (best single model): R@1 drops from 40.9 → 29.9 on AudioCaps.
  • MCQ-Neg Negation accuracy (AudioCaps): 0.2–1.1% across all models vs. 25% random baseline; far below Affirmation accuracy of 67.4–85.4%.
  • Hybrid MCQ accuracy (AudioCaps): 14.0–51.3%, intermediate between Affirmation and Negation types.
  • Steering (λ=0.2) on MCQ-Neg: WAVE-7B improves from 43.5% → 54.5% (AudioCaps) and 42.2% → 52.4% (Clotho); CLAP checkpoints gain only +0.1 to +3.5 pp.
  • Steering on Retrieval-Neg R@5: Average gains of +1.2 pp (AudioCaps) and +1.7 pp (Clotho)—marginal.
  • Text-side diagnosis (5 LAION-CLAP checkpoints): Bag-of-concepts similarity to negated queries = 0.82; negated-query margin = +0.018 vs. affirmative margin = +0.072, confirming negated embeddings remain attracted to the negated concept.

Limitations

  • No negation-aware training method is proposed or evaluated; the paper diagnoses the problem and tests only a lightweight inference-time fix.
  • Steering requires knowing which concept is negated at inference time, limiting applicability to structured queries.
  • NegEval-Audio currently covers only AudioCaps and Clotho; generalization to other acoustic domains (music, speech-centric) is untested at scale.
  • Evaluation relies on Qwen2.5-Omni-7B for audio-side verification; errors in this model propagate to benchmark construction, despite the 96.7% human agreement.
  • The paper's text-side diagnosis is limited to five LAION-CLAP checkpoints on ESC-50 controlled stimuli and may not fully capture failure modes in MLLM-based models like WAVE.

Relevance to Vision-Language Models

This paper directly parallels well-established findings in VLM compositionality research, where contrastive models like CLIP similarly conflate "a dog" and "no dog" due to bag-of-words-like matching in the joint embedding space. The NegEval-Audio construction pipeline (concept extraction → plausibility-constrained negative proposal → modality-side verification) provides a transferable methodology that could be adapted to probing negation in image-language models. The training-free steering approach (\(e^* = e_C - \lambda e_{C_{\text{neg}}}\)) mirrors techniques explored in VLM negation correction, and the finding that it fixes local ranking but not global geometry is a directly comparable insight. Researchers tracking VLMs should view this as cross-modal evidence that affirmation bias is a systemic property of contrastive embedding training, not an audio-specific artifact, strengthening the case for negation-aware contrastive objectives across modalities.