Skip to content

Structured-Condensed Prompt Tuning in Vision-Language Models for Fine-grained Image Recognition

🕒 Published (v1): 2026-07-07 12:09 UTC · Source: Arxiv · link

Why this paper was selected

Structured prompt tuning for CLIP fine-grained recognition — practical, reproducible PEFT recipe for VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Existing VLM prompt tuning methods treat class labels as independent tokens, ignoring inter-class semantic structure critical for fine-grained image recognition. SCPT addresses this by injecting a compact topology-preserving class-relation encoding into the prompt and suppressing noisy supervision via SVD-based spectral filtering. Evaluated across 14 fine-grained benchmarks, SCPT achieves state-of-the-art results in both few-shot and base-to-novel generalization settings.

Problem

CLIP-based prompt tuning (CoOp and variants) assigns each class a discrete, isolated token, discarding the rich hierarchical and inter-class semantic relationships that are essential for distinguishing visually similar subcategories. This bottleneck is especially severe in fine-grained image recognition (FGIR) where category semantics are narrow and overlapping, and where coarse supervision can actively mislead optimization.

Method

SCPT builds on the CoOp textual-prompt framework with two modules:

Semantic Relation Encoding (SRE). CLIP's frozen text encoder generates per-class embeddings \(X = \{X_i\}_{i=1}^N\). A pairwise cosine similarity matrix \(S \in \mathbb{R}^{N \times N}\) is computed as \(S_{ij} = (X_i \cdot X_j) / (\|X_i\|\|X_j\|)\). To fit within CLIP's token-length constraint, random projection via the Johnson–Lindenstrauss lemma compresses \(S\) to a binary code \(P = \text{sign}(SW^\top) \in \{0,1\}^{N \times d}\) where \(W \in \mathbb{R}^{d \times N}\) is drawn i.i.d. from \(\mathcal{N}(0,1)\) and \(d = \lceil \log_2(N) + d_{\text{free}} \rceil\). Each row \(P_i\) is word-embedded to \(R_i = e(P_i)\) and prepended to the learnable context vectors: \(V_i = \{R_i, v_1, \ldots, v_m\}\).

Semantic Condensation Loss (ScLoss). Handcrafted-prompt embeddings \(X \in \mathbb{R}^{N \times M}\) are denoised via truncated SVD, retaining top-\(K\) singular components to form \(X' = U_K \Sigma_K V_K^\top\). \(K\) is selected adaptively using a probabilistic gate derived from the Marchenko–Pastur spectral bound: singular values below threshold \(\tau\) are retained with probability \(e^{-(σ_i - \tau)^2 N_{\text{below}}}\), where \(N_{\text{below}}\) is the count of sub-threshold values. The regularization loss is \(\mathcal{L}_{\text{sc}} = \frac{1}{N} \sum_{i=1}^N \|Y_i - X'_i\|_2^2\), pulling learnable embeddings toward denoised general semantics. The total objective is \(\mathcal{L} = \mathcal{L}_{\text{ce}} + \lambda \mathcal{L}_{\text{sc}}\) with \(\lambda = 8.0\).

Key Contributions

  • SCPT framework: structure-aware textual prompt tuning that replaces isolated class tokens with topology-preserving relational embeddings
  • SRE: signed random projection of the CLIP inter-class similarity matrix into a compact binary Hamming-space encoding that preserves pairwise semantic distances within CLIP's token budget
  • ScLoss: SVD-based spectral denoising of handcrafted-prompt supervision with a Marchenko–Pastur-informed adaptive \(K\) selector to suppress noise while retaining discriminative semantics
  • State-of-the-art on 14 FGIR benchmarks in both 16-shot and base-to-novel generalization settings using only textual prompt tuning (visual encoder untouched)

Results

  • 16-shot average accuracy across 14 datasets: 76.70%, outperforming TCP (CVPR 2024) on 13 of 14 benchmarks; underperforms only on Oxford Pets (the dataset with the fewest fine-grained categories)
  • Baselines compared: CLIP zero-shot, CoOp (IJCV 2022), CoCoOp (CVPR 2022), KgCoOp (CVPR 2023), TCP (CVPR 2024), ProText (AAAI 2025), ATPrompt (ICCV 2025)
  • Datasets include Dog Breed, Oxford Pets, Oxford Flowers, Stanford Cars, Web Cars, Stanford Dogs, Fruit92, CUB200, Food101, Food172, Food200, Food500, FGVC Aircraft, Veg200
  • SVD denoising with adaptive \(K\) consistently improves over fixed-\(K\) or no-denoising baselines (see Figure 3)

Limitations

  • The SRE binary encoding is computed offline from CLIP's pretrained embeddings; if CLIP's pretraining encodes inaccurate taxonomy for a domain, the relation matrix inherits those errors
  • Performance degrades on Oxford Pets relative to TCP, suggesting the inter-class topology encoding provides diminishing benefit when the number of fine-grained categories is small
  • Adaptive \(K\) selection is grounded in an asymptotic Gaussian-noise assumption (Marchenko–Pastur); finite-sample deviations may make the threshold \(\tau = 0.3\) suboptimal across domains
  • The paper truncates before reporting full base-to-novel generalization numbers in the excerpted text; numerical margins over baselines on novel classes are not fully visible
  • Textual-only prompt tuning means visual-side domain gaps are not addressed

Relevance to Vision-Language Models

SCPT directly addresses a structural blind spot in CoOp-style VLM adaptation: the loss of inter-class semantic topology when class names are embedded as independent tokens. The SRE component is a lightweight, annotation-free technique for injecting CLIP's own latent taxonomy back into the prompting pipeline, relevant to anyone using CLIP or similar contrastive VLMs for fine-grained downstream tasks. The ScLoss approach—applying Marchenko–Pastur spectral filtering to remove noise from handcrafted-prompt supervision—is broadly applicable as a regularization strategy in any alignment-based prompt tuning method. Together these techniques extend the prompt tuning literature toward richer semantic grounding without modifying the vision encoder, making them composable with visual prompting and adapter methods in the broader VLM adaptation ecosystem.