Skip to content

Fine-grained CLIP fine-tuning with self-annotated region alignment

🕒 Published (v1): 2026-07-15 10:06 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

SFF-CLIP is a fine-tuning method that improves CLIP's region-level (dense) representations using only the image-text pairs already present in training data — no external region proposals, pre-defined category lists, or auxiliary models required. It generates region-phrase pairs at run-time via text-specific gradient heat maps (Grad-ECLIP) and aligns them with a weighted contrastive loss, while a momentum model preserves CLIP's original image-level matching ability.

Problem

CLIP's contrastive pre-training optimizes the [cls] token against a whole-caption embedding, leaving patch-token features poorly calibrated for localized region-language matching. Existing fine-grained fine-tuning methods (RegionCLIP, CLIPSelf, FineCLIP, CLIM) address this but require offline region proposals, pre-defined concept pools, or auxiliary VLMs to generate region annotations — adding data preparation cost, limiting semantic diversity, and often degrading global image-text retrieval performance.

Method

SFF-CLIP fine-tunes EVA-CLIP (ViT-B/16 or ViT-L/14) with two jointly trained objectives:

Self-annotated fine-grained alignment. At each training step, NLTK parses the input caption using an "adjective + noun" rule to extract up to \(N=5\) concept phrases \(\{p_t\}\). For each phrase, a text-specific spatial heat map \(H_{ti}\) is computed in-line via a gradient-based method (Grad-ECLIP): $\(H_{ti} = \sum_c g_c \cdot u_i \cdot v_i\)$ where \(g_c\) are channel-wise gradient weights, \(u_i = \Phi(\cos(q_\text{cls}, k_i))\) is a loosened attention weight, and \(v_i\) are dense patch-token value embeddings extracted by modifying the last ViT layer to drop self-attention. Region embeddings \(F_{r_t} = \sum_{hw} H_t \cdot F_d\) are obtained by spatially weighting the dense feature map. A weighted cross-entropy-style loss penalizes mismatches: $\(L_{fg} = -\sum_t w_t \left[(1-S(F_{r_t},F_{p_t}))^2 \log S(F_{r_t},F_{p_t}) + \sum_{t'\neq t} S(F_{r_t},F_{p_{t'}})^2 \log(1-S(F_{r_t},F_{p_{t'}}))\right]\)$ The weight \(w_t = \max(H_t)\) down-weights phrases with no visual correlate (e.g., "Tuesday").

Momentum global contrastive loss. A momentum copy \(\hat{M}\) of the model (updated as \(\hat{M} \leftarrow (1-\alpha)\hat{M} + \alpha M\), \(\alpha=0.005\)) produces per-sample match scores that re-weight the standard CLIP InfoNCE loss \(L_\text{contrastive}\), focusing training on well-matched pairs and preserving global representation. The total loss is \(L = L_{fg} + L_\text{contrastive}\).

Fine-tuning uses the MS COCO train2017 images with captions; no region bounding boxes or labels are consumed.

Key Contributions

  • Run-time self-annotation: region-phrase pairs are generated on-the-fly from image-text pairs alone — no offline detector, concept pool, or auxiliary VLM.
  • Grad-ECLIP-guided region aggregation: repurposes an XAI saliency method as a training signal to localize concept phrases in the dense feature map.
  • Phrase-matching weight \(w_t = \max(H_t)\): automatically suppresses visually ungrounded phrases without manual filtering.
  • Momentum global alignment: EMA model re-weights the image-text contrastive loss to prevent degradation of CLIP's zero-shot image-level capability during dense fine-tuning.
  • Demonstration that fine-grained gains are achievable on ViT-based CLIP without CNN backbones or external annotation pipelines.

Results

  • Zero-shot region classification (ADE20K panoptic boxes, ViT-B/16 Top-1): SFF-CLIP 29.4 vs. CLIP 18.6, RegionCLIP 27.9, CLIPSelf 27.9, FineCLIP 27.0, CLIM 25.7.
  • Zero-shot region classification (MS COCO panoptic boxes, ViT-B/16 Top-1): SFF-CLIP 62.2 vs. CLIP 41.4, CLIPSelf 60.9, RegionCLIP 59.4, FineCLIP 57.7, CLIM 53.4.
  • Zero-shot region classification (MS COCO panoptic masks, ViT-B/16 Top-1): SFF-CLIP 52.1 vs. CLIP 30.6, CLIPSelf 49.0, RegionCLIP 47.4, FineCLIP 48.0.
  • ViT-L/14 (MS COCO panoptic boxes Top-1): SFF-CLIP 75.2 vs. CLIP 58.1 (+17.1 pp).
  • SFF-CLIP requires no region proposals (R.P.), region labels (R.L.), or pre-defined categories (P.C.), unlike RegionCLIP and FineCLIP.
  • Global image-text retrieval performance is maintained (quantitative table referenced but truncated in provided text).

Limitations

  • Phrase extraction via NLTK's "adjective + noun" rule is heuristic and may miss complex noun phrases or fail on non-standard captions.
  • Heat map quality is bounded by the CLIP model's own attention at the time of generation, which is noisy early in training.
  • Training data is restricted to MS COCO train2017 images (≈118k images) with captions — a relatively small corpus; scalability to web-scale crawled pairs is asserted but not demonstrated.
  • The method inherits CLIP's visual vocabulary: concepts absent from CLIP's pretraining remain poorly grounded.
  • The maximum of \(N=5\) phrases per sentence is a hard cap that may truncate densely described scenes.
  • Code is not yet released.

Relevance to Vision-Language Models

SFF-CLIP directly addresses a well-known structural weakness of dual-encoder VLMs — the misalignment between patch-token features and language — through a lightweight, annotation-free fine-tuning recipe that applies to any ViT-based CLIP variant. The technique of repurposing XAI gradient maps as training-time region supervisors is a novel alternative to the dominant paradigm of offline proposal generation, and it preserves global contrastive alignment via a momentum model rather than treating it as a trade-off. For researchers tracking VLMs, this work is relevant both as a practical recipe for enabling open-vocabulary dense prediction without heavy data infrastructure, and as a signal that explainability tools (Grad-CAM-style methods) can serve as self-supervisory signals inside VLM training loops.