Skip to content

Aligning Visual Contrastive learning models via Preference Optimization

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper adapts Preference Optimization (PO) methods—DPO, IPO, and KTO—to non-generative contrastive vision-language models like CLIP, framing contrastive learning as a one-step MDP. The approach trains CLIP to prefer semantically correct outputs over adversarial or biased ones while preserving pretrained knowledge via KL-regularization. It demonstrates effectiveness on typographic attack robustness and gender bias mitigation.

Problem

CLIP-like contrastive VLMs are vulnerable to typographic attacks (misleading text in images causes misclassification) and encode societal biases (e.g., gender bias). While PO alignment methods (RLHF, DPO) have been applied to generative LLMs, they had not been extended to contrastive learning models, leaving a systematic alignment strategy absent for this class of models.

Method

Contrastive learning is cast as a one-step MDP: the input is the state, the retrieved text/image label is the action, and the policy is the softmax over CLIP similarity scores. A preference dataset D_pref contains triples (x, y_w, y_l) where y_w is the correct label and y_l is the adversarial/biased label. Three PO objectives—DPO, IPO, KTO—are adapted to this contrastive policy. Under a frozen text encoder, the policy ratio simplifies to (1/τ)(I_θ(x) − I_ref(x))^T (T(y_w) − T(y_l)), meaning the image encoder is pushed toward the preferred text embedding difference. A KL-divergence regularization loss L_reg over a clean dataset D_reg prevents forgetting. Additionally, a learnable linear head W = UΣV^T is learned on top of the image encoder; post-training, singular values are scaled as Σ → Σ^t to continuously interpolate between task behaviors (e.g., OCR vs. object detection, or flipped vs. neutral gender understanding) without retraining.

Key Contributions

  • First application of DPO, IPO, and KTO preference optimization to non-generative contrastive learning models (CLIP).
  • Formal MDP framing of contrastive retrieval, enabling direct import of the PO mathematical machinery.
  • Lemma showing the DPO/IPO policy ratio decomposes into a geometrically interpretable image-encoder update direction.
  • SVD-based linear head with post-training singular-value scaling (W_t = UΣ^t V^T) for continuous, training-free control over learned concepts.
  • Joint loss combining preference optimization and KL-regularization to retain pretrained knowledge.

Results

  • Typographic attack robustness (Table 1, 8 datasets, KTO vs. best baseline): improvements of up to +19.63 accuracy points on typographic images (FGVCAircraft), with average improvement of +5.43 on typographic splits and only −0.78 average drop on clean splits.
  • KTO is the best-performing PO variant overall; DPO and IPO also outperform baselines on typographic accuracy across most datasets.
  • KL-divergence from reference (Figure 5a): all three PO methods yield lower KL divergence than cross-entropy fine-tuning, confirming better pretrained knowledge retention.
  • Gender concept control (Figure 3): the SVD scaling parameter t continuously interpolates from original gender predictions to reversed gender predictions while keeping task-detection accuracy stable.
  • Image retrieval (Figure 4): reversed-gender model retrieves 6W/4M for "police" vs. 2W/8M for vanilla CLIP; neutralized model (t=t*) retrieves 5W/5M.

Limitations

  • Only evaluated on CLIP; generalization to other contrastive architectures (e.g., ALIGN, SigLIP) is unverified.
  • Preference datasets D_pref are typically small and may not cover the full data manifold, motivating but not fully solving the need for regularization.
  • The theoretical analysis (Lemma 3.1) assumes a frozen text encoder, which may not hold in all fine-tuning settings.
  • Typographic attack training uses ImageNet-100, so cross-dataset generalization is limited by the domain of D_pref.
  • Gender-flipping is a demonstration of concept control rather than a debiasing solution; the ethical implications of concept reversal are not analyzed.

Relevance to Vision-Language Models

This paper directly addresses a known robustness gap in CLIP-style VLMs by importing the alignment toolbox from generative LLMs—a previously unexplored transfer. For researchers tracking VLMs, it establishes that the MDP framing of contrastive retrieval is mathematically valid and practically useful, and that PO methods preserve pretrained representations better than cross-entropy fine-tuning. The SVD-based singular-value steering is a lightweight, interpretable mechanism for post-hoc concept control in CLIP embeddings, relevant to any work on VLM bias mitigation, adversarial robustness, or concept disentanglement.