Conditional Representation Learning for Customized Tasks

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Universal visual representations learned by CLIP and similar VLMs are dominated by a single semantic axis (typically "category" or "shape"), making them suboptimal for downstream tasks that require alternative criteria (e.g., "color", "texture", "count"). Conditional Representation Learning (CRL) reframes this as a basis transformation: an LLM generates descriptive texts spanning the user-specified criterion, a VLM encodes them as a text basis, and image representations are projected onto that basis via a simple dot product. The result is a training-free, criterion-aware representation that consistently outperforms standard VLM embeddings across classification, clustering, and retrieval.

Problem

Existing representation learning methods—including large VLMs like CLIP, ALIGN, and MetaCLIP—produce universal embeddings that collapse onto the most salient semantic dimension of their training data (typically object category or shape). When a downstream task cares about a different criterion (e.g., scene, color, fabric), these universal representations are suboptimal, and the standard remedy (supervised fine-tuning) is expensive and requires labeled data for each new criterion.

Method

CRL operates in two stages, both training-free at inference:

1. Basis construction: Given a user-specified criterion C, an LLM is prompted with "Generate common expressions to describe the C, as many as possible." The resulting descriptive words W are encoded by the VLM text encoder using the prompt template "Objects with the C of W", producing a normalized text matrix T ∈ ℝ^{d×dim} where each row is a text embedding of one descriptor.

2. Representation transformation: Image representations I (from the VLM image encoder) are projected into the conditional feature space via R = IT⊤. Each element of the resulting conditional representation R corresponds to the image's cosine similarity with a specific descriptor, making the representation both criterion-aligned and interpretable. The alignment between image and text in VLM joint embedding space is what makes this projection semantically meaningful.

The approach requires only a pre-trained VLM (ViT-B/32 or ViT-B/16 CLIP backbone in experiments) and an LLM; no gradient updates are needed.

Key Contributions

• Formalizes the problem of criterion-specific representation learning and frames it as a basis transformation in VLM joint embedding space.
• A training-free method (CRL) that adapts any VLM image representation to an arbitrary user-specified criterion at virtually zero overhead.
• Demonstrates generality across four task families: few-shot classification, clustering, similarity retrieval, and fashion retrieval, all under diverse non-dominant criteria.
• Provides an interpretable representation where each dimension has an explicit semantic meaning tied to a generated descriptor.

Results

• Customized few-shot learning (Clevr4-10k, Cards): CLIP+CRL achieves ~10% mean accuracy gain over vanilla CLIP across criteria; on the "color" criterion at 1-shot, improvement reaches ~+40%.
• Customized clustering (Clevr4-10k): CLIP+CRL obtains >75% ACC boost over CLIP on the "color" criterion; mean NMI improves from 26.61 to 59.49 when combined with CLIP.
• Customized similarity retrieval (GeneCIS): CLIP+CRL reaches 26.8% mean recall vs. 19.9% for CLIPimage+text and 17.9% for LinCIR/CIG; with Combiner fine-tuned weights, 32.6% vs. 28.9%.
• Customized fashion retrieval (DeepFashion): training-free CLIP+CRL achieves 7.93 mean MAP vs. 6.08 for CLIP; after MLP fine-tuning, 11.33 mean MAP, surpassing the previous SOTA RPF (10.22) by ~10% relatively.
• CRL performance is stable for textual basis sizes ≥ ~100 descriptors; performance degrades only when the number of descriptive texts is very small.

Limitations

• Underperforms task-specialized clustering methods (CC, SCAN) on the universal "shape" criterion, which those methods were specifically designed for.
• The text basis is only an approximation of the true semantic basis; strategies to filter or refine LLM-generated descriptors that generalize across all criteria have not been found.
• Performance depends on LLM output quality; no robust basis-filtering procedure is provided.
• Evaluated only on CLIP-family VLMs (CLIP, ALIGN, MetaCLIP, BLIP2); generalization to other VLM architectures is assumed but not verified.

Relevance to Vision-Language Models

CRL directly exploits the joint image-text embedding space of VLMs as a semantic routing mechanism, treating the text encoder as a projective basis generator rather than a classifier head—a novel use of VLM alignment. It exposes a systematic bias in CLIP-style pretraining (dominance of category semantics) and proposes a lightweight post-hoc correction requiring no fine-tuning of the VLM itself. For researchers tracking VLMs, this is relevant as an example of zero-shot adaptation of frozen VLMs to non-standard semantic axes, complementing prompt-tuning and adapter-based approaches. It also connects to the interpretability-through-text-decomposition line of work (e.g., Text-Span on CLIP's attention heads) by showing that LLM-generated descriptor sets can substitute for hand-crafted or learned bases.