Skip to content

Enhancing Compositional Reasoning in CLIP via Reconstruction and Alignment of Text Descriptions

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

READ (REconstruction and Alignment of text Descriptions) is a CLIP fine-tuning method that adds two auxiliary objectives—token-level reconstruction via a frozen decoder and sentence-level paraphrase alignment—to improve compositional reasoning. READ-CLIP achieves state-of-the-art average accuracy of 64.1% across five compositional benchmarks, outperforming the prior best (FSC-CLIP) by up to 4.1%.

Problem

CLIP-style contrastive training causes the text encoder to focus on individual object-referring words rather than inter-word relational structure, producing embeddings that are insensitive to compositional changes (e.g., swapping subject/object in a sentence). Existing fixes either rely solely on hard negatives—which can induce shortcut learning—or introduce auxiliary objectives only for the image encoder, leaving the text encoder bottleneck unaddressed.

Method

READ augments the standard contrastive loss with two text-encoder-targeted objectives:

  1. Token-level reconstruction loss: Given an original caption \(T_i\), the text encoder produces embedding \(v_i\), which is linearly projected (\(h_i = W^\top v_i\)) and fed to a frozen pretrained T5 decoder. The decoder is supervised to reconstruct an alternative (paraphrased) caption \(y_i^{(k)}\), not the original—this choice prevents overfitting to exact wording and forces the encoder to capture relational structure needed to generalize across phrasings.

  2. Sentence-level alignment loss: LLM-generated paraphrases \(T_i'\) of each caption are treated as positives; other batch items' paraphrases are negatives. An InfoNCE-style loss directly aligns paraphrase pairs in the shared embedding space, enforcing semantic consistency across surface-level variation.

The final loss is \(\mathcal{L}_\text{READ} = \mathcal{L}_\text{Contrastive} + \alpha\,\mathcal{L}_\text{Token Recon} + \beta\,\mathcal{L}_\text{Sentence Align}\), fine-tuned on 100K MS-COCO samples for 5 epochs with ViT-B/32.

Key Contributions

  • Two complementary text-encoder auxiliary objectives (token-level reconstruction, sentence-level alignment) that address compositional reasoning at different granularities.
  • Empirical demonstration that reconstructing an alternative caption (rather than the original) mitigates overfitting and improves generalization over training epochs.
  • READ is modular: applying it on top of NegCLIP and FSC-CLIP yields consistent gains (+1.0% and +0.8% average) without hyperparameter re-tuning.
  • Ablation isolating complementarity: reconstruction primarily lifts WhatsUp (+3.1%), VALSE (+1.9%), CREPE (+3.2%); alignment primarily lifts SugarCrepe++ ITT/TOT.

Results

  • READ-CLIP achieves 64.1% average accuracy across WhatsUp, VALSE, CREPE, SugarCrepe, SugarCrepe++ (ITT+TOT).
  • Outperforms pretrained CLIP by +12.1%, NegCLIP by +4.5%, FSC-CLIP (prior best) by +1.7% on average.
  • Benchmark-level highlights: WhatsUp 43.9% (+2.4 vs FSC-CLIP), VALSE 76.2% (+1.8), CREPE 41.5% (−1.0 vs FSC-CLIP but +0.8 avg), SugarCrepe 87.0% (+1.8), SugarCrepe++ ITT 69.8% (+1.9), TOT 66.2% (+1.8).
  • When applied to vanilla CLIP (no hard negatives), READ improves average from 57.0% → 58.2%.
  • Performance stable across wide hyperparameter ranges; effective with K=1 target sequence and T5-Large decoder.
  • 10–20% paraphrase noise degrades average accuracy by only 1.2–2.0%.

Limitations

  • Requires multiple captions per image; single-caption datasets need LLM-generated paraphrases, adding complexity and dependency on LLM quality.
  • Reconstruction decoder fixed to T5; other generative architectures (e.g., GPT-style) not explored.
  • Decoder weights are frozen throughout training; fine-tuning the decoder's effect on compositional reasoning is unexplored.
  • Evaluation confined to ViT-B/32 and MS-COCO fine-tuning regime; scaling behavior with larger backbones or datasets is unverified.

Relevance to Vision-Language Models

READ directly addresses a well-documented failure mode of contrastive VLMs—bag-of-words text representations—by introducing principled text-encoder-specific training signals without modifying the visual side, making it a clean diagnostic contribution. The finding that token-level reconstruction of alternative captions outperforms self-reconstruction is a practically important insight for encoder training design. As a drop-in fine-tuning method compatible with NegCLIP and FSC-CLIP, READ is immediately applicable to any CLIP-family model and relevant to downstream tasks requiring precise relational understanding (VQA, referring expression, composed image retrieval). It reinforces the emerging consensus that compositional reasoning requires explicit structural supervision beyond standard contrastive alignment.