Skip to content

Unlabeled Data Improves Fine-Grained Image Zero-shot Classification with Multimodal LLMs

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

AutoSEP is a fully unsupervised, black-box prompt learning framework that iteratively optimizes a description-generation prompt for MLLMs using only unlabeled images. By framing prompt quality as an instance-level image-description retrieval score (analogous to contrastive SSL), it guides MLLMs to articulate fine-grained discriminative features without any labeled data or model fine-tuning. It achieves +13% average accuracy over vanilla zero-shot and +3% over the best competing unsupervised baseline across 8 fine-grained tasks.

Problem

MLLMs perform well on general zero-shot classification but degrade on fine-grained tasks (distinguishing bird subspecies, plant varieties, dog breeds) because they overlook subtle visual differences without explicit guidance. Prior fixes require large labeled datasets and white-box model access—both impractical for fine-grained domains where annotation is expensive and model weights may be proprietary.

Method

AutoSEP introduces an intermediate textual description step: a learnable "description-generation prompt" p conditions the MLLM to produce a structured description t = M(x, p), which is concatenated with the image for final classification Å· = M(x, [q, t]).

The prompt is optimized via an instance-level classification scoring function Ψ̂(X, p, k): for each unlabeled image xᵢ, the MLLM is asked whether xᵢ's generated description or a randomly sampled negative description xⱼ better matches the image. The score aggregates binary correctness V(xᵢ, xⱼ) over all image/negative pairs (with random ordering to cancel positional bias), approximated with k random negatives per image.

Prompt refinement follows a Reflect-Modify loop: error pairs where V(xáµ¢, xâ±¼)=0 are sampled; the MLLM reflects on why the current prompt caused those errors, then proposes revised prompts. Multiple candidate prompts are scored each iteration and the top b are carried forward for N total iterations. Only black-box MLLM API calls are required; no gradients, no labeled data.

Key Contributions

  • AutoSEP: first framework to improve MLLM zero-shot fine-grained classification using only unlabeled data and black-box API access.
  • Instance-level classification as an unsupervised optimization signal, derived from contrastive SSL intuition but operating purely in text space via binary description-retrieval queries.
  • Iterative Reflect-Modify prompt optimization that improves prompt diversity and discriminative specificity over iterations.
  • Empirical demonstration across 8 subsets from CUB-200, iNaturalist, Stanford Dogs, and VegFru on three MLLMs (Gemini 1.5 Flash, GPT-4o, Qwen2-VL-72B).

Results

  • +13% average accuracy over vanilla zero-shot across all 8 tasks and 3 MLLMs.
  • +3% average over best competing unsupervised baseline.
  • Selected task highlights (AutoSEP vs. best baseline accuracy %):
  • CUB_cuckoo / GPT-4o: 75.12 vs. 68.29 (vanilla zero-shot)
  • CUB_vireo / GPT-4o: 87.42 vs. 87.19 (majority-vote baseline, marginal but consistent)
  • iNat_butterfly / Qwen2-VL: 55.38 vs. 51.50 (majority vote)
  • StanfordDogs_terrier / GPT-4o: 92.18 vs. 91.42 (vanilla)
  • StanfordDogs_terrier / Qwen2-VL: 81.70 vs. 77.74 (majority vote)
  • VegFru_greens / Qwen2-VL: 72.22 vs. 57.78 (SPO baseline)
  • Pearson correlation between instance-level and class-wise accuracy: 0.47–0.84 across datasets/models, validating the surrogate signal.
  • Reliable with as few as 60 unlabeled samples; one iteration on 60 images with Gemini 1.5 Flash takes ~12 minutes.
  • SPO (a text-only prompt optimizer adapted to multimodal) consistently underperforms AutoSEP, confirming that pairwise output comparison is a weak signal for visual description quality.

Limitations

  • Assumes fine-grained distinctions can be verbalized in natural language; fails for distinctions too subtle or non-linguistic to describe.
  • Requires a sufficient number of unlabeled images (~60+) for stable optimization; small pools introduce noise.
  • Lags behind fully supervised methods that have access to class-level label guidance.
  • Computational cost scales as O(60n) MLLM queries per iteration; larger datasets or more iterations increase API costs.
  • Instance-level scoring measures inter-image distinctiveness of descriptions, not direct task accuracy, creating a proxy gap.

Relevance to Vision-Language Models

AutoSEP directly addresses a known failure mode of MLLMs—insufficient attention to fine-grained visual features in zero-shot settings—through prompt engineering rather than weight updates, making it model-agnostic and immediately applicable to closed-source VLMs like GPT-4o and Gemini. The framework reframes contrastive self-supervised learning as a prompt optimization objective in text space, a novel bridge between SSL theory and black-box VLM adaptation. For researchers tracking VLMs, it establishes that unlabeled test-time data is a tractable signal for improving zero-shot performance without any architectural changes, which has implications for test-time adaptation, domain shift, and low-resource deployment of VLMs. The instance-level retrieval scoring mechanism is a transferable design pattern for other unsupervised VLM evaluation tasks.