TULIP: Token-length Upgraded CLIP

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TULIP replaces CLIP's fixed absolute positional encodings with RoPE-based relative encodings via a two-step adaptation (distillation then fine-tuning), enabling the text encoder to handle captions of arbitrary length beyond the 77-token hard limit. This avoids retraining from scratch while improving long-caption cross-modal retrieval and text-to-image generation.

Problem

CLIP and similar contrastive VLMs use absolute positional encodings fixed to 77 tokens, causing truncation of long captions and preventing the model from capturing pairwise token relationships over longer distances. Prior work (Long-CLIP) only interpolates existing absolute encodings, which partially addresses the hard limit but does not fundamentally improve relational token modeling or generalize to arbitrary lengths.

Method

TULIP replaces CLIP's absolute positional encodings with Rotary Position Embeddings (RoPE) in the text encoder's self-attention layers, then adapts the model in two stages:

1. Relative Position Distillation: The original CLIP text encoder acts as teacher; a student encoder initialized with RoPE is trained on short captions (≤77 tokens) using cosine similarity loss to transfer alignment knowledge without multimodal retraining.
2. Relative Position Expansion: The distilled student is fine-tuned on full-length captions from 1M image-caption pairs (ShareGPT4V) using NTK-aware scaled RoPE (α=8.0) and a joint contrastive loss that balances short (L_short) and long (L_long) caption objectives: L_total = λ·L_short + (1−λ)·L_long.

At inference, the model supports arbitrary token lengths (evaluated at 248 tokens to match Long-CLIP for fair comparison).

Key Contributions

• First contrastive VLM with relative positional encodings (RoPE) for long-caption understanding.
• Generalizable two-phase adaptation (distillation + expansion) applicable to any CLIP-like model without retraining from scratch.
• Long-DCI benchmark: 7,000 human-annotated image–caption pairs averaging 200 tokens/caption, addressing diversity and saturation limitations of existing long-caption retrieval benchmarks.
• Demonstrated improvements on cross-modal retrieval (long and short) and plug-in text encoder replacement for text-to-image generation (SDXL).

Results

Long-caption retrieval (Long-DCI, ViT-L-14): - TULIP Img2Txt: 55.7 vs. Long-CLIP 54.0, CLIP 35.0 - TULIP Txt2Img: 56.4 vs. Long-CLIP 46.1, CLIP 37.0

Long-caption retrieval (Urban-1K, ViT-L-14): - TULIP Img2Txt: 90.1 vs. Long-CLIP 82.7, CLIP 68.7 - TULIP Txt2Img: 91.1 vs. Long-CLIP 86.1, CLIP 52.8

Short-caption retrieval (Flickr30k, ViT-L-14): - TULIP Img2Txt R@1: 56.7 vs. Long-CLIP 53.4, CLIP 48.5 (competitive without short-caption-specific tuning)

Ablation — positional encoding type (Long-DCI Img2Txt, ViT-L-14): - RoPE: 55.7, CoPE: 50.8, Absolute: 41.9

Text-to-image generation: SDXL + TULIP correctly renders fine-grained details beyond 77-token boundary (e.g., spatial relations, attributes) where SDXL + CLIP, Long-CLIP, PIXART-Alpha, and ELLA fail; evaluated qualitatively with human evaluation in appendix.

Limitations

• Training data dependency: performance bounded by quality of GPT-4V-generated ShareGPT4V captions.
• Practical token length constrained by the average caption length in training data (~174 tokens), causing diminishing returns beyond 231–308 tokens despite theoretical unboundedness of RoPE.
• Text-to-image evaluation is qualitative only; no quantitative generation metrics reported in the main paper.
• Evaluation at 248 tokens chosen to match Long-CLIP rather than to maximize TULIP's potential range.

Relevance to Vision-Language Models

TULIP directly targets a structural bottleneck in contrastive VLMs (CLIP's 77-token limit) by importing NLP context-extension techniques—RoPE and NTK-aware scaling—into the vision-language domain, a transfer that was previously underexplored. The distillation-first approach makes this adaptation lightweight and applicable to any CLIP-like backbone, lowering the barrier for the broader community to upgrade existing checkpoints. For downstream tasks that increasingly rely on detailed image descriptions (dense captioning, retrieval with rich queries, text-to-image with complex prompts), TULIP provides a drop-in text encoder improvement. The Long-DCI benchmark also fills a measurement gap, offering a more reliable evaluation surface for future long-caption VLM work.