Skip to content

Attributes Should Come from Images, Not Class Names: Distribution-Conditioned Attribute Selection for Vision-Language Models

🕒 Published (v1): 2026-07-21 04:34 UTC · Source: Arxiv · link

Why this paper was selected

Deepak Pathak + Zhiqiu Lin; LLM-generated CLIP descriptors carry no visual evidence — fundamental rethink

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

LLM-generated class descriptors for CLIP-based zero-shot classification carry almost no visual evidence themselves — removing the class name from the prompt collapses ImageNet accuracy from 59.5% to 15.5%. This paper diagnoses the failure as conditioning on class names rather than images, and proposes distribution-conditioned attribute selection: scoring a large, fixed attribute pool against the target image collection in CLIP's joint embedding space, then keeping top-ranked attributes per class. The image-conditioned attributes recover substantial accuracy, generalize across distribution shifts, and enable interpretable few-shot classification faster than prompt tuning.

Problem

Descriptor-augmented CLIP classifiers (e.g., Menon & Vondrick) generate per-class attributes using an LLM conditioned on the class name alone, never consulting any image. The resulting descriptors describe the canonical concept (e.g., "strawberries are red") but fail under distribution shift (e.g., ImageNet-Sketch strawberries are colorless line drawings). The class-name confound means the apparent accuracy gains are attributable to prompt ensembling around the class name, not to genuine attribute-level visual evidence.

Method

Given a frozen CLIP model and a fixed pool of \(m\) attribute strings \(\{a_1,\ldots,a_m\}\), each image \(I\) is mapped to an attribute feature vector via scaled cosine similarities:

\[s_j = \tau \cdot \cos(f_{\text{img}}(I),\, f_{\text{txt}}(a_j)), \quad j = 1,\ldots,m,\quad \tau=100\]

Attributes are encoded bare (no class name, no template) to avoid reintroducing the class-name confound. Class-attribute weights \(w_{c,j}\) are computed as either (a) the per-class mean of \(s_j\) over labeled images, or (b) the weights of a linear probe trained on the attribute features. The top-\(k\) attributes per class form the prompt set \(A_c\). At inference, a test image is classified by:

\[\hat{c} = \arg\max_c \frac{1}{|A_c|} \sum_{j \in A_c} \omega_{c,j} \cdot \cos(f_{\text{img}}(I), f_{\text{txt}}(a_j))\]

The attribute pool is drawn from VAW, LSA, and GPT-3 descriptors (~634 VAW attributes used in main experiments). Both CLIP encoders remain frozen throughout; the only fitted object is the linear ranking of the pool.

Key Contributions

  • Class-name confound established: LLM descriptors stripped of the class name collapse from 59.5% to 15.5% on ImageNet, demonstrating that prior gains were not attribute-level evidence.
  • Distribution-conditioned attribute selection: image-conditioned scoring of a fixed pool, with a matched-pool ablation that isolates the selection mechanism as the cause of gains (not vocabulary differences).
  • Strong extreme few-shot learner: with 1 image per class, the method reaches 60.1% on ImageNet vs. 57.2% for CoOp, fitting in under 1 minute vs. 14 hours, while producing a human-readable word list.
  • Interpretable distribution shift description: subtracting mean attribute profiles of two datasets yields ranked word lists describing the shift (e.g., ImageNet-Sketch gains "colorless", "gray", "cartoon").
  • Extensions: attribute prompts handle unparseable scientific names (iNaturalist: 3% → 7.25%), and domain-level rising attributes generate useful prompt templates (up to +3.7 points on ImageNet-Sketch).

Results

  • Attribute-only (no class name), top-5, VAW+LSA pool: 23.8% on ImageNet vs. 15.5% for LLM descriptors — a 53% relative improvement; gains hold on all four shifted variants (largest margin: ImageNet-R, 28.2% vs. 17.9%).
  • Matched-pool ablation: reselecting from the identical GPT pool lifts 15.5% → 19.4%, attributing the 3.9-point gain purely to the selection mechanism.
  • Uncapped attribute budget (16 images/class, weighted, VAW+LSA): 45.5% attribute-only ImageNet accuracy, nearly 3× the LLM-descriptor baseline.
  • With class name, top-100 attributes: 60.8% vs. 59.5% for LLM descriptors and 55.3% for class-name-only CLIP (CLIP-RN50).
  • Few-shot (1-shot): 60.1% vs. CoOp 57.2%, WiSE-FT 58.3%, fit time <1 min vs. 14 hr; crossover with gradient methods occurs at ~8 shots.
  • Color-shift (fruit dataset): CLIP zero-shot drops 30 points (81.0% → 51.3%); attribute prompts improve both (85.6% / 55.1%) but do not close the gap.
  • Domain prompt templates: top-5 rising domain attributes as templates add up to +3.7 points on ImageNet-Sketch over the plain class-name baseline.

Limitations

  • Attribute-only prompts remain far below class-name prompting in absolute terms; top-5 attributes per class cannot fully separate 1,000 ImageNet categories.
  • Accuracy plateaus beyond 4 shots because the linear probe only reranks a fixed pool — no capacity increase from additional images.
  • The VAW/LSA attribute pool is largely human-centric; fine-grained or scientific domains (e.g., iNaturalist species) are poorly served by a general-purpose vocabulary.
  • Color shift from counter-stereotypical appearances is reduced but not repaired; robustly handling such cases remains open.
  • Results reported with CLIP-RN50 only in most tables; generalization to ViT-based or larger CLIP backbones is not fully characterized.

Relevance to Vision-Language Models

This paper directly challenges a widely-used prompting paradigm for VLMs by exposing that the standard LLM-descriptor pipeline conflates prompt ensembling with attribute-level visual grounding — a methodological clarification with broad implications for how CLIP-based classifiers are evaluated and interpreted. The distribution-conditioned selection framework offers a principled, training-free mechanism for grounding textual attributes in a specific image distribution, which is relevant to any VLM application where the deployment distribution differs from the pretraining assumption. The few-shot results show that semantic attribute spaces can serve as an efficient intermediate representation, competitive with learned soft prompts at extreme low-shot regimes. The dataset-description-in-words capability suggests a new use case for frozen VLMs in continual learning and domain adaptation monitoring.