Skip to content

When Negation Is a Geometry Problem in Vision-Language Models

🕒 Published (v1): 2026-03-20 23:06 UTC · Source: Arxiv · Venue: CVPR · link

Why this paper was selected

CVPR; negation failure in CLIP reframed as geometry problem with novel fix

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CLIP models collapse affirmative and negated text queries into indistinguishable embeddings, and prior fine-tuning approaches suffer from false negatives in evaluation. This paper demonstrates that a linear "negation direction" already exists in CLIP's text encoder hidden states, and that test-time representation steering along this direction improves negation-aware retrieval without any fine-tuning. The authors also propose an MLLM-as-a-judge evaluation framework that better isolates negation understanding from retrieval correctness.

Problem

CLIP treats language as a bag of words, making negation cues (e.g., "no logos", "not on the grass") effectively invisible—affirmative and negated queries yield nearly identical embeddings. Existing data-centric fixes (fine-tuning on synthetic negated captions) are undermined by: (1) massive false negatives introduced by single-ground-truth annotation over databases with many valid matches, and (2) model collapse in some baselines (e.g., ConCLIP maps all negated queries to a single point). Standard Recall@K metrics conflate retrieval quality with negation understanding and cannot detect these failure modes.

Method

MLLM-as-a-judge evaluation: Given a negated query \(c_n\), an LLM generates two yes/no questions: \(q_r\) (semantic correctness: "is there a dog?") and \(q_n\) (negation satisfaction: "is the dog running?"). A Qwen3-VL model answers sequentially; only images passing \(q_r\) are tested on \(q_n\). This is applied over a 25K-image database, 5× larger than existing benchmarks.

Negation direction discovery: 4,000 COCO captions are negated by an LLM. The \(\langle\text{eos}\rangle\) token hidden states \(h_l \in \mathbb{R}^d\) are extracted from each layer \(l\) of the CLIP text encoder. A linear binary classifier (L-BFGS, no bias) is trained to separate original vs. negated captions, achieving ≥99% test accuracy at layer 4 across ViT-B/32, ViT-B/16, and ViT-L/14.

Representation steering: The classifier weight vector \(W^l\) is unit-normalized to \(W^l_\text{dir} = W^l / \|W^l\|\). At inference, every layer's hidden state is steered: $\(h^l = (1 - \alpha)h^l + \alpha W^l_\text{dir} \|h^l\|\)$ where \(\alpha = 0.13\) (tuned by ablation) controls steering strength. The norm \(\|h^l\|\) is preserved to avoid pushing representations out-of-distribution. Steering is applied to all layers.

Key Contributions

  • Identifies two fundamental flaws in existing negation evaluation: false negatives from single-ground-truth annotation and model collapse masked by Recall@K.
  • Proposes MLLM-as-a-judge (Qwen3-VL-32B/8B/4B) as a dual-question evaluation protocol separating semantic retrieval from negation satisfaction.
  • Demonstrates via linear probing that a negation direction exists in CLIP text encoder hidden states with ≥99% separability at layer 4.
  • Introduces test-time representation steering that surpasses all fine-tuned baselines using only 4K text-only samples (0.03% of data used by CLIP-CC12M).
  • Curates two controlled benchmarks: SimpleNeg (900 concise negated queries, 25K-image database) and N-COCO (200 images, 10 uncommon negated scenes) for distribution-shift evaluation.

Results

NegBench (ViT-B/32, MLLM-as-a-judge, Qwen3-VL-32B): - CLIP baseline: Top-1 Retrieval+Negation = 40.3, Avg.5 = 25.4 - NegCLIP (best fine-tuned baseline): Top-1 = 53.1, Avg.5 = 32.1 - ConCLIP collapses completely (Fig. 3); CLIP-CC12M marginal over CLIP

SimpleNeg (ViT-B/32, Qwen3-VL-32B): - CLIP baseline: Top-1 = 44.9, Top-5 = 84.7 - CLIP-CC12M (12M training samples): Top-1 = 53.1, Top-5 = 88.0 - Steering (ours, 4K samples, no fine-tuning): Top-1 = 54.3, Top-5 = 98.0; outperforms all baselines

N-COCO out-of-distribution benchmark (Recall@1): - CLIP: 0.60; NegCLIP: 0.50; CLIP-CC12M: 0.50; ConCLIP: 0.00 - Steering (ours): 0.80 — fine-tuned models degrade under distribution shift, steering generalizes

Limitations

  • Evaluated only on ViT-B/32 backbone for comparisons with all baselines (NegCLIP and CLIP-CC12M unavailable for larger variants).
  • Steering strength \(\alpha\) requires tuning; overly large \(\alpha\) distorts semantic similarity and degrades performance.
  • PCA visualization shows affirmative and negated embeddings remain only partially separable in 3D; full geometric separation is not achieved.
  • SimpleNeg is deliberately simple (≤2 objects/actions per query) to enable controlled analysis; performance on complex multi-clause negation queries is not the primary focus.
  • MLLM-as-a-judge relies on proprietary/large open-source models (Qwen3-VL-32B), which may not be accessible in all deployment settings.
  • N-COCO uses synthetically generated images (GPT-5 Image), which may not fully capture real-world distribution shift.

Relevance to Vision-Language Models

This paper directly addresses a known structural weakness of CLIP-style joint embedding VLMs—their inability to encode logical negation—and reframes it as a geometry problem in the representation space rather than a data problem. The finding that a linearly separable negation direction exists at ≥99% accuracy inside CLIP's text encoder challenges the prevailing view that negation is simply absent from the model and must be injected via synthetic fine-tuning data. For researchers tracking VLMs, the representation engineering / steering approach is broadly applicable: it suggests that other underexpressed linguistic properties (spatial relations, attribute binding, cardinality) may similarly have latent directional structure that can be exploited at inference time without weight updates. The MLLM-as-a-judge evaluation framework also offers a reusable paradigm for assessing fine-grained language understanding in retrieval settings where false-negative annotation is endemic.