VITRIX-CLIPIN: Enhancing Fine-Grained Visual Understanding in CLIP via Instruction-Editing Data and Long Captions

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

CLIP-IN is a two-stage fine-tuning framework that improves CLIP's fine-grained visual understanding by repurposing instruction-editing datasets (image manipulation data) as hard negative image-text pairs, combined with long descriptive captions processed via a RoPE-adapted text encoder. On the MMVP benchmark, CLIP-IN raises OpenAI ViT-L/14 accuracy from 18.5% to 30.4% and, when used as the visual backbone in LLaVA-1.5, reduces hallucinations and improves multimodal reasoning across standard MLLM benchmarks.

Problem

Standard CLIP suffers from two compounding weaknesses: (1) global image-text alignment objectives overlook subtle local visual differences (color, spatial relations, object state), producing "CLIP-blind spots" measurable on MMVP; (2) the fixed 77-token absolute positional encoding in the text encoder prevents the use of longer, richer captions. Existing fixes—region-level contrastive learning, self-distillation, synthetic hard negatives via T2I models, and positional interpolation—each address only one axis, often at the cost of general zero-shot performance.

Method

Stage 1 — RoPE text encoder via knowledge distillation. Absolute positional embeddings in the CLIP text encoder are replaced with Rotary Positional Encodings (RoPE). A student encoder (RoPE) is distilled from a frozen teacher (original CLIP) using cosine-similarity loss on truncated long captions, preserving short-text capability while enabling variable-length input.

Stage 2 — Joint contrastive training on two complementary data sources:

• Instruction editing hard negatives: Each UltraEdit tuple (source image \(I^s\), target image \(I^t\), source caption \(T^s\), target caption \(T^t\)) yields hard negative pairs \((I^s, T^t)\) and \((I^t, T^s)\). A symmetric hard negative contrastive loss \(L_{HN}\) penalizes both image-to-text and text-to-image mismatches by augmenting the InfoNCE denominator with these cross-paired negatives, pushing the model to distinguish fine-grained visual-semantic differences in both directions.
• Long captions: InternVL2-generated captions (~300 tokens) are used with standard InfoNCE. Original short alt-text captions are retained to preserve zero-shot classification performance.

Overall loss: \(L = \lambda_{short} L_{CL}(X, Y_{short}) + \lambda_{long} L_{CL}(X, Y_{long}) + \lambda_{HN} L_{HN}\), with \(\lambda_{short}=1.0\), \(\lambda_{long}=0.1\), \(\lambda_{HN}=0.1\).

Training uses ~4M UltraEdit samples and ~18M long-caption pairs; 16× A100-80GB GPUs, AdamW, lr=1e-4.

Key Contributions

• Repurposing instruction-editing datasets (originally for image generation) as high-quality, visually grounded hard negative sources for contrastive CLIP training.
• Symmetric hard negative contrastive loss operating in both image→text and text→image directions—enabled by the controlled nature of editing data (unlike T2I-synthesized negatives).
• RoPE integration into CLIP's text encoder via knowledge distillation, extending context capacity without full retraining and outperforming positional interpolation (LongCLIP baseline).
• Demonstration that the two data sources are complementary: only their combination improves MMVP in the downstream MLLM setting (+3.3 pp over baseline), while each alone slightly degrades it.

Results

MMVP (fine-grained visual perception): - ViT-L/14 @ 224: 18.5% → 30.4% (+11.9 pp) vs. OpenAI CLIP - ViT-L/14 @ 336: 20.0% → 26.1% (+6.1 pp) vs. OpenAI CLIP; surpasses DIVA (25.2%) - SigLIP2 ViT-SO/14 @ 224: 35.6% → 36.3% (+0.7 pp)

Zero-shot retrieval (Flickr30K + MS-COCO average): - ViT-L/14 @ 224: 60.7% → 72.9% (+12.2 pp) vs. OpenAI CLIP - SigLIP2 ViT-SO/14 @ 224: 76.4% → 78.8% (+2.4 pp)

Long-caption retrieval (DCI + ShareGPT4V average): - SigLIP2 ViT-SO/14 @ 224: 62.0% → 67.3% (+5.3 pp)

Compositional benchmarks (vs. OpenAI CLIP, ViT-L/14 @ 224): - SugarCrepe: 73.8% → 79.4% avg (+5.6 pp) - Winoground text: 28.3% → 33.0% (+4.7 pp) - SPEC: 32.0% → 34.8% (+2.8 pp)

LLaVA-1.5 MLLM (ViT-L/14 @ 336, Vicuna-7B): - MMVP: 24.7 → 28.0 (+3.3 pp) - MME-Perception: 1510.7 → 1709.0 - MMBench-EN: 64.3 → 72.9; MMBench-CN: 58.3 → 70.3 - Surpasses DIVA on all reported MLLM benchmarks

RoPE vs. LongCLIP interpolation ablation (ViT-L/14 @ 224): - IN-1K: 70.1% → 76.3% (+6.2 pp); short retrieval avg: 69.4% → 72.9% (+3.5 pp)

UltraEdit vs. TripletData (1.4M samples, ViT-L/14 @ 224): - Average retrieval: 70.58% → 72.9% (+2.32 pp) in favor of UltraEdit

Limitations

• Long-caption retrieval (DCI+ShareGPT4V avg 76.4%) still trails FG-CLIP (81.8%), primarily because FG-CLIP trains on ~1 billion long-caption pairs vs. 18 million here.
• Each data source alone (instruction editing or long captions) individually decreases MMVP performance in the LLaVA-1.5 setting; the improvement requires their combination, suggesting fragile interdependency.
• MMVP gains on already-strong backbones (DFN-H, SigLIP2) are marginal or zero in some categories, indicating diminishing returns on higher-capacity base models.
• RoPE distillation is performed only on long captions; no evaluation of intermediate context lengths or domain shift between distillation data and downstream long-text tasks.
• Evaluation is confined to CLIP as a frozen visual encoder plug-in (LLaVA-1.5); no end-to-end MLLM pretraining experiments.

Relevance to Vision-Language Models

This paper directly addresses a fundamental bottleneck in VLM pipelines: CLIP visual encoders inherit fine-grained blindspots that propagate into hallucinations in downstream MLLMs such as LLaVA. The instruction-editing hard negative strategy is a practically significant finding—it converts a large existing data modality (image editing datasets) into a free source of controlled, visually grounded negatives without requiring expensive T2I synthesis. The RoPE-distillation approach for long-text CLIP encoders is directly relevant to any VLM that relies on richer textual descriptions at the vision-encoder level, complementing work like LongCLIP and FineCLIP. The demonstrated transfer: better CLIP → reduced hallucination in MLLMs is a concrete causal link that matters for practitioners choosing or fine-tuning visual backbones.