Skip to content

Advancing Compositional Awareness in CLIP with Efficient Fine-Tuning

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CLIC (Compositionally-aware Learning in CLIP) is an efficient fine-tuning method that improves compositional reasoning in CLIP models by concatenating image pairs and generating multiple positives plus hard negatives from their captions. Unlike prior work that improved lexical sensitivity at the cost of semantic understanding and retrieval, CLIC achieves consistent gains on both SugarCrepe++ and downstream retrieval benchmarks. Applied to CLIPS, it yields state-of-the-art compositional ITT scores while further improving an already SOTA retrieval model.

Problem

CLIP models exhibit "bag of words" behavior, failing to understand relational and compositional structure between concepts. Prior fine-tuning methods (NegCLIP, DAC, TripletCLIP, SVLC) improve performance on SugarCrepe—which only tests lexical sensitivity—but regress on SugarCrepe++, which requires genuine semantic understanding (assigning high similarity to both a positive P1 and a lexically different but semantically equivalent P2 against a hard negative N). Additionally, most prior methods degrade downstream retrieval and classification performance.

Method

CLIC fine-tunes only the text encoder (vision encoder frozen) using concatenated image pairs as training inputs. For each batch, two images are horizontally concatenated into a single composite image. Five captions are generated from their original dense captions: - p1: concatenation of first sentences from each image (high-level description) - p2: sentence-shuffled version of p1 (teaches order invariance) - p3, p4: random additional sentence pairs from each caption (reinforces part-level matching) - n: hard negative constructed by swapping one word between the two sentences using spaCy, changing meaning without changing vocabulary distribution

Training uses three losses: (1) extended contrastive loss over all four positives (Eq. 4); (2) S-Neg loss pushing each positive away from the hard negative (Eq. 5); (3) uni-modal L2 loss between p1 and p2 embeddings to enforce syntactic invariance (Eq. 6). Every second iteration alternates standard single-image CLIP contrastive training to prevent concatenation-induced distribution shift. No LLM hard-negative generation or synthetic image generation is required. High-quality dense captions come from PixelProse (RedCaps, CC12M) or CogVLM-recaptioned Laion (~1M samples).

Key Contributions

  • Efficient fine-tuning method requiring no LLM or diffusion model for hard-negative generation; combinatorial image-pair space grows quadratically with dataset size
  • Consistent SugarCrepe++ improvements across architectures (ViT-B/32, ViT-B/16, ViT-L/14) and pre-training regimes (CLIP, CLIPA, CLIPS)
  • CLIPS + CLIC achieves SOTA ITT scores on SugarCrepe++ among all CLIP-family models with +9% average ITT improvement, while further improving CLIPS's already-SOTA retrieval by +1.3% text / +2.2% image on MS-COCO
  • Replacing LLaVA-1.5-7b's CLIP vision encoder with CLIC improves compositional reasoning (WinoGround text +6.8%, group +3.0%) while maintaining GQA, TextVQA, SQA-I scores and improving MME perception (1441.0 → 1465.3)

Results

SugarCrepe++ ITT (ViT-B/32, Replace/Swap): - CLIP baseline: 69.5 / 45.7 - CLIC-RedCaps: 76.0 / 61.5 (best among same-architecture models) - NegCLIP: 70.5 / 56.4; DAC-LLM: 53.7 / 32.2 (both regress on SugarCrepe++)

CLIPS + CLIC (ViT-L/14, SugarCrepe++ Replace ITT / Swap ITT): - CLIPS baseline: 79.3 / 60.6; CLIPS + CLIC-RedCaps: 84.9 / 75.1 (+5.6 / +14.5 pp)

WinoGround (CLIPS + CLIC): - Text / Image / Group: 41.7 / 17.5 / 15.2 vs. CLIPS: 36.2 / 16.0 / 12.5 — best CLIP-family scores

Retrieval (MS-COCO R@5, CLIC-CC12M vs. CLIP baseline): - Text retrieval: 76.9 vs. 74.1; Image retrieval: 60.8 vs. 54.6

Zero-shot classification: minimal degradation (CLIC-LAION: 61.7% IMNET vs. 63.3% CLIP; CLIC-CC12M: 62.2%)

LLaVA-1.5 (CLIC vs. CLIP encoder): - SugarCrepe++ Swap-Att ITT: 71.2 vs. 69.8; WinoGround Text: 43.5 vs. 36.7; MME-P: 1465.3 vs. 1441.0

Limitations

  • Hard-negative generation relies on spaCy word swapping, which can accidentally produce valid descriptions if swapped words share meaning; no constraint is imposed on word position or semantic plausibility
  • Fine-tuning only the text encoder (for efficiency) may limit gains in vision-side compositionality; full model fine-tuning is only explored for LLaVA integration
  • SugarCrepe++ TOT (text-only) scores do not improve uniformly across datasets (CLIC-LAION improves Replace TOT to 60.1 but degrades Swap TOT vs. baseline)
  • Evaluation is limited to concatenated-image benchmarks and standard retrieval; compositional reasoning over natural single images is not directly assessed post-fine-tuning
  • LLaVA integration results are preliminary and the authors explicitly defer a detailed study to future work

Relevance to Vision-Language Models

CLIC directly addresses a core diagnostic failure of contrastive VLMs: that high SugarCrepe scores reflect lexical heuristics rather than true semantic compositionality. By demonstrating that SugarCrepe++ is the appropriate benchmark—and that existing methods regress on it—the paper re-frames the standard evaluation target for the field. The image-concatenation trick for generating semantically grounded hard negatives and multiple positives is a lightweight, architecture-agnostic recipe applicable to any CLIP-family model, making it immediately relevant to practitioners building retrieval or grounding systems on top of CLIP encoders. The LLaVA experiment provides early evidence that compositionally fine-tuned encoders transfer benefits into generative VLMs without catastrophic forgetting, connecting to the broader challenge of building VLMs with genuine visual relational reasoning.