Skip to content

Seeing What’s Not There: Negation Understanding Needs More Than Training

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; shows negation understanding requires more than training data scaling

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CLIP-based VLMs exhibit "affirmation bias," ignoring negation words and treating "a cat without a dog" as semantically equivalent to "a cat and a dog." This paper proposes a training-free, zero-shot embedding correction that projects out the negated concept's semantic direction from the full caption embedding using vector arithmetic, achieving state-of-the-art negation understanding without any fine-tuning.

Problem

CLIP and its fine-tuned variants (NegCLIP, ConCLIP) fail to represent linguistic negation: negation words ("no," "not," "without") are effectively ignored, causing the model to treat negated captions nearly identically to affirmative ones in embedding space. Data-centric fixes (curating hard-negative datasets and retraining) improve NegBench scores but cause catastrophic forgetting of general zero-shot capabilities, and negated embeddings from different categories remain tightly clustered rather than semantically distinguished.

Method

Given a caption \(C\) containing negation, the method operates entirely in CLIP text embedding space:

  1. Rule-based negation scope detection: A hand-crafted parser scans for a corpus of negators (explicit: "no/not/never"; absence indicators: "devoid/absent/empty"; contracted forms: "n't/can't"). Forward-scoped pre-negators (e.g., "devoid of") and backward-scoped post-negators (e.g., "absent") are handled separately; scope extends to the nearest clause boundary. This extracts the negated concept string \(C^{\text{neg}}\).

  2. Anchor embedding: A bias vector \(a\) is computed as the mean CLIP encoding of semantically neutral words (e.g., "neutral," "balanced"), capturing non-informative embedding bias that must be reintroduced after subtraction.

  3. Embedding correction: Let \(e_c = \text{CLIP}(C)\), \(e^{\text{neg}} = \text{CLIP}(C^{\text{neg}})\). The corrected embedding is:

\[e^* = e_c - \lambda \frac{\langle e_c, e^{\text{neg}} \rangle}{\langle e^{\text{neg}}, e^{\text{neg}} \rangle} \left(e^{\text{neg}} - a\right)\]

The fraction \(\frac{\langle e_c, e^{\text{neg}} \rangle}{\langle e^{\text{neg}}, e^{\text{neg}} \rangle} e^{\text{neg}}\) is the projection of \(e_c\) onto \(e^{\text{neg}}\) (the directional offset to remove); \(\lambda\) is a scalar hyperparameter tuned on 5% of COCO-MCQ. Positive (non-negated) captions pass through unchanged.

Key Contributions

  • Demonstrates that CLIP text embeddings support compositional arithmetic: \(e_{cf} \approx e_c + e_f\) (cosine similarity 0.86), and \(e_{cf} - e_f + a \approx e_c\) (cosine similarity 0.83), establishing the geometric foundation for the correction.
  • Proposes the projection-based embedding correction (Eq. 2) that removes only the semantically shared component of the negated concept from the full caption embedding, avoiding over-subtraction.
  • Shows fine-tuning on negation datasets degrades zero-shot classification (ImageNet, Caltech101, Oxford Pets, etc.) by increasing inter-class confusion—a concrete cost of data-centric approaches.
  • Achieves SOTA on NegBench MCQ without any training, outperforming models fine-tuned on CC12M-NegFull.
  • Demonstrates generalization across SigLIP, SigLIP2, AlignCLIP, and TripletCLIP backbones.

Results

All results on NegBench (ViT-B/32); MCQ accuracy and R-Neg@5 retrieval reported:

  • CLIP (baseline) MCQ: COCO 24.7% → 72.5% (+47.8%), VOC2007 24.3% → 78.6% (+54.3%), MSRvtt 27.5% → 50.0% (+22.5%) — outperforms CLIP and NegCLIP (both trained on CC12M-NegFull).
  • CLIP retrieval R-Neg@5: COCO 49.7% → 63.2% (+5.9%), MSRvtt 44.5% → 49.0% (+5.5%).
  • NegCLIP* + Ours MCQ: COCO 54.03% → 66.22% (+13.3%), VOC2007 → 75.1% (+15.4%), MSRvtt → 54.1% (+7.9%); retrieval COCO 59.25% → 67.9% (+0.9%).
  • SigLIP MCQ: COCO 30.8% → 61.15% (+32.25%), VOC2007 30.7% → 66.9% (+36.1%), MSRvtt → 46.8% (+16.1%).
  • VALSE Existence: CLIP 71.0% → 79.5% (+8.5%), vs. NegationCLIP baseline 78.3%.
  • Flickr30K human-annotated retrieval: Acc@R5 67.4% → 70.4% (+3.0%), Acc@R1 34.0% → 41.3% (+7.3%).

Limitations

  • The \(\lambda\) hyperparameter requires validation data (5% of COCO-MCQ) for tuning; performance may degrade on out-of-distribution negation styles without retuning.
  • Rule-based negation scope detection is fragile for complex syntactic structures (nested clauses, long-range dependencies, pragmatic negation) not covered by the hand-crafted grammar.
  • Only CLIP-family joint-embedding models are addressed; generative VLMs (e.g., LLaVA, GPT-4V) with decoder architectures are out of scope.
  • Improvement on retrieval R-Neg@5 is notably smaller than on MCQ (~6% vs ~50%), suggesting that coarse retrieval and fine-grained MCQ benefit asymmetrically.
  • The anchor embedding (mean of neutral words) is a heuristic; its quality and stability across domains are not systematically analyzed.

Relevance to Vision-Language Models

This paper directly targets a known structural failure mode of contrastive VLMs: the bag-of-words bias induced by contrastive pretraining that renders negation invisible. The work demonstrates that CLIP's embedding space encodes enough compositional geometry to support post-hoc semantic surgery without retraining, challenging the dominant assumption that negation requires dataset-level fixes. For researchers tracking VLMs, this connects to the broader thread on compositional understanding (ARO, SugarCREPE, CREPE) and offers a lightweight inference-time remedy applicable to any CLIP-based pipeline (retrieval, text-to-image generation, referring segmentation). The finding that fine-tuning for negation degrades general zero-shot capabilities is a concrete empirical warning against specialization at the cost of representational breadth.