Skip to content

SuperCLIP: CLIP with Simple Classification Supervision

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SuperCLIP augments CLIP's contrastive training objective with a token-level classification loss by treating raw text subword tokens (IDF-weighted K-hot vectors) as classification targets for the vision encoder via a single lightweight linear head. This costs only 0.077% additional FLOPs and requires no extra annotated data, yet consistently improves zero-shot classification, image-text retrieval, and purely visual tasks across model scales.

Problem

CLIP's contrastive objective optimizes only global image-text similarity, ignoring token-level semantic signals. Consequently: (1) fine-grained attributes (object states, spatial relations, actions) are underweighted because they rarely co-occur within the same batch; (2) richer re-captioned data (e.g., Recap-DataComp-1B) actually hurts standard CLIP performance rather than helping it; (3) performance degrades sharply under small batch sizes since contrastive learning depends on large batches for sufficient negatives.

Method

SuperCLIP adds a single linear classification head on top of the vision encoder's averaged image features. Each caption is tokenized with CLIP's subword tokenizer (vocabulary size V) and converted to a V-dimensional K-hot vector; tokens are then reweighted by IDF (w_c = log(|D| / (1 + df(c)))) to suppress stopwords and common terms. The normalized weighted label distribution ŷ_c becomes the classification target, trained with cross-entropy. The total loss is L_Total = L_CLIP + λ · L_Class (λ ≥ 1.0 recommended). No changes to the text encoder, no additional annotated data, no region proposals—just one linear layer added to the image encoder. Under the "Dual" caption setting, the contrastive loss uses short original captions while the classification loss uses long re-captioned texts, decoupling coarse and fine-grained supervision.

Key Contributions

  • Identifies that CLIP's contrastive paradigm cannot exploit fine-grained token-level supervision even when rich captions are available, and quantifies this with co-occurrence statistics and word-image similarity analysis.
  • Proposes a trivially simple fix: IDF-weighted subword token classification on vision encoder features, adding only 0.051 GFLOPs (0.077% overhead) to ViT-L.
  • Demonstrates the Dual-caption training scheme that extracts coarse semantics from short captions (contrastive) and fine-grained semantics from long captions (classification) without ratio hyperparameter search.
  • Shows the method generalizes to SigLIP and FLIP (SuperSigLIP, SuperFLIP) and improves LLaVA-1.5 downstream benchmarks when used as the vision encoder.
  • Demonstrates the classification loss mitigates CLIP's batch-size sensitivity, maintaining performance at batch sizes as small as 1K.

Results

  • Zero-shot ImageNet (B-512M): SuperCLIP 63.5% vs. CLIP 60.5% (+3.0%); (L-512M): 70.1% vs. 66.1% (+4.0%); (L-12.8B): 80.0% vs. 79.0% (+1.0%).
  • Flickr30K image retrieval (L-512M, R@1): 62.4% vs. 57.0% (+5.4%).
  • Re-captioned data (100% long captions, L-512M): SuperCLIP-L 33.8% vs. CLIP-L 30.0% avg. across 38 datasets (+3.8%); CLIP drops to 30.0% from 45.7% (original captions), while SuperCLIP recovers to 33.8%.
  • Dual-caption (L-512M): 49.5% avg. across 38 datasets vs. CLIP 46.8% (+2.7%), outperforming the carefully tuned 0.8/0.2 CLIP mixture (46.8%).
  • SuperSigLIP (B-512M): +3.7% ImageNet, +2.9% Flickr retrieval over SigLIP baseline.
  • SuperFLIP (B-512M): +3.4% ImageNet, +5.3% Flickr text retrieval over FLIP baseline.
  • Purely visual tasks (B-512M): Linear probing ImageNet +1.5%, PASCAL segmentation +7.7 mIoU, ADE20K +4.1 mIoU, NYUv2 depth -0.022 RMSE.
  • LLaVA-1.5 (VQAv2): 69.6% vs. 67.8%; MMBench: 55.9% vs. 49.1%.

Limitations

  • The classification head supervises only the vision encoder; no symmetric improvement to the text encoder is explored (noted as future work).
  • IDF computation requires a full pass over the training corpus to compute document frequencies; scalability to trillion-scale datasets is unaddressed.
  • The method is validated only up to ViT-L scale with 12.8B training samples; behavior at ViT-H/G scales or with significantly larger datasets is untested.
  • The "Dual" caption setting requires access to both original and re-captioned data simultaneously, adding a data preparation dependency.
  • Ablations use only 256M samples with ViT-B/16; optimal λ may shift at larger scales.

Relevance to Vision-Language Models

SuperCLIP directly addresses a known failure mode of contrastive VLM training—its insensitivity to token-level semantics—with a nearly zero-cost fix that benefits the entire CLIP-family ecosystem (SigLIP, FLIP). The finding that richer captions hurt standard CLIP contradicts a common assumption in the community and has implications for data-centric VLM scaling strategies. The Dual-caption training scheme is a practical template for combining short web captions with MLLM-generated re-captions without expensive hyperparameter search. Improved visual encoders also flow downstream into MLLMs (LLaVA-1.5), making this relevant to anyone building on CLIP-initialized multimodal systems.