No Hard Negatives Required: Concept Centric Learning Leads to Compositionality without Degrading Zero-shot Capabilities of Contrastive Models

🕒 Published (v1): 2026-03-26 17:58 UTC · Source: Arxiv · Venue: CVPR 2026 · link

Why this paper was selected

CVPR 2026; concept-centric learning enables compositionality without zero-shot degradation

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Contrastive vision-language models (e.g., CLIP/SigLIP) fail at compositional reasoning because long captions permit bag-of-words shortcuts and global pooling destroys attribute-noun binding. C²LIP fine-tunes SigLIP using only noun-phrase sub-captions and a parameter-free cross-modal attention-pooling mechanism, reaching SOTA on compositionality benchmarks without hard negatives and without degrading zero-shot or retrieval performance.

Problem

Existing contrastive V&L models learn bag-of-words (BoW) representations because: (1) long, detailed captions are retrievable without compositional understanding; (2) global pooling in both encoders erases the spatial/semantic binding between attributes and nouns before any compositional signal can be learned. Hard-negative fine-tuning patches surface symptoms but over-fits to narrow benchmark distributions and consistently hurts zero-shot classification and retrieval.

Method

C²LIP (Concept-centric CLIP) fine-tunes SigLIP on CC3M with three contrastive objectives combined as: $\(L_\text{total} = L_\text{contrastive} + \lambda_\text{npc} L_\text{npc} + \lambda_\text{xac} L_\text{xac}\)$

Noun-phrase extraction. spaCy dependency parsing extracts noun-phrase concepts \(\{c_k\}\) from each caption offline.

Noun-phrase concept loss (\(L_\text{npc}\)). Each noun-phrase embedding \(c_k\) (pooled from text-encoder token representations of that span) is contrasted against the global visual embedding \(v\) using a multi-positive SigLIP sigmoid loss, forcing \(v\) to encode all concepts in the scene.

Cross-attended concept-aware loss (\(L_\text{xac}\)). A parameter-free cross-modal attention-pooling extracts a concept-specific visual embedding: $\(\hat{v}(c) = \bar{V}^{\prime\top} \cdot \mathrm{attn}(c,\, \bar{V}^\prime)\)$ where \(\bar{V}' = f_\text{MLP}(\bar{V})\) reuses the SigLIP value projection. \(\hat{v}(c_k)\) is aligned to \(c_k\) via sigmoid loss. Because the attention pooling is parameter-free, gradients flow directly into the penultimate visual representation, encouraging binding before global pooling. The cross-attention module is dropped at inference, leaving the pipeline identical to vanilla SigLIP.

Key Contributions

• Hard-negative-free fine-tuning pipeline that improves compositionality using only real data positives (noun-phrases from existing captions).
• Parameter-free cross-modal attention-pooling that injects binding signal pre-pooling without adding inference cost or trainable parameters.
• Multi-positive SigLIP loss variant (\(L_\text{npc}\)) that aligns a single image to all its noun-phrase concepts simultaneously.
• Empirical demonstration that compositionality improvements do not require sacrificing zero-shot classification or retrieval, unlike prior hard-negative methods.

Results

• SugarCrepe (ViT-B, CC3M fine-tuned): C²LIP achieves 94.2 / 88.3 / 73.1 (Replace/Swap/Add) vs. SigLIP baseline 87.9 / 85.6 / 69.7; best among composition-aware ViT-B models on the overall average (75.0 vs. next best ~68.2).
• SugarCrepe++ (more robust benchmark): C²LIP 44.2 (Replace-TOT) and 79.8 (Replace-I2T); outperforms all composition-aware baselines on attribute-binding average (75.7 vs. best baseline 67.4 from NegCLIP).
• Retrieval: Improves over fine-tuned SigLIP baseline on Flickr30k (97.0 vs. 95.6 R@5) and MSCOCO (82.7 vs. 80.3 R@5).
• ImageNet zero-shot: Minor drop vs. fine-tuned SigLIP baseline (79.7 → 79.7; small degradation relative to original SigLIP 86.5), substantially smaller than most composition-aware methods.
• Attribute-binding (Tab. 3): C²LIP average 75.7 vs. best competing method NegCLIP 67.4, DAC-LLM 54.8, CLIC 63.7.
• SOTA on SugarCrepe and SugarCrepe++ among ViT-B-scale models.

Limitations

• Fine-tuned on CC3M (3M samples), a narrower domain than SigLIP's WebLI pretraining; the domain shift partially explains the ImageNet drop.
• Training drives a more scene-centric representation, which hurts highly object-centric tasks like ImageNet classification.
• Noun-phrase extraction via spaCy dependency parsing may fail on syntactically complex or noisy captions.
• Applicability to non-SigLIP architectures (e.g., standard CLIP with CLS pooling) is claimed in principle but not empirically validated.
• Cross-modal attention pooling is dropped at inference, so the richer concept-grounded representations are never directly accessible during deployment.

Relevance to Vision-Language Models

This work directly addresses a well-known structural failure of CLIP-family models—BoW collapse and binding loss through global pooling—with a clean, zero-inference-cost remedy. For researchers tracking VLMs, C²LIP demonstrates that architectural interventions at the pooling stage (rather than data augmentation with hard negatives) can yield more generalizable compositional representations. The result challenges the dominant hard-negative paradigm and suggests that compositionality and zero-shot generalization are not fundamentally in tension, which has broad implications for how future contrastive VLMs are trained and evaluated. The noun-phrase-as-positive framing is directly compatible with existing pretraining pipelines, making it a practical drop-in technique.