U-shaped Multi-granularity Learning for Vision-Language Models¶

🕒 Published (v1): 2026-07-16 13:18 UTC · Source: Arxiv · link

Why this paper was selected

Prompt learning granularity dilemma for VLMs; practical recipe for cross-task generalization

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

UPrompt is a U-Net-inspired prompt learning framework for VLMs that resolves the granularity dilemma between global context and local detail by constructing parallel multi-granularity hierarchies in both visual and textual modalities, connected via bidirectional information flow. A coarse-to-fine cascaded enhancement injects global context into fine-grained features via cross-attention, while fine-to-coarse hierarchical supervision via knowledge distillation ensures semantic consistency across scales. Evaluated on 17 benchmarks, it achieves state-of-the-art on cross-modal retrieval, few-shot classification, and base-to-novel generalization.

Problem¶

Existing prompt learning methods for VLMs operate at a single fixed granularity, creating an irreconcilable trade-off: global prompts (e.g., CoOp) lack fine-grained semantic awareness and underperform on fine-grained datasets (−11.31% vs. TAP on FGVCAircraft), while local/fine-grained prompts lose global context and inter-region compositional relationships. Multi-granularity methods (TAP, HiCroPL) that do exist treat granularity levels as independent modules fused only at the final stage, missing cross-scale dependencies and bidirectional information flow.

Method¶

UPrompt constructs \(K\) parallel granularity pathways in both modalities, extending CLIP encoders \(F\) and \(G\) to multi-granularity versions \(\{F^{(k)}, G^{(k)}\}_{k=1}^{K}\) (coarsest to finest).

Visual hierarchy: Progressive spatial pooling on patch embeddings reduces the \(14\times14\) finest representation to \(7\times7\), \(4\times4\), and \(1\times1\) tokens; learnable prompts \(P_v^{(k)} \in \mathbb{R}^{M \times d}\) are concatenated at each level.

Textual hierarchy: An LLM (Llama 3-8B) constructs nested semantic representations via iterative enrichment — coarsest level extracts core semantics \(\Phi_\text{abstract}(t)\); finer levels append granularity-specific elaborations \(\Phi_\text{refine}^{(k)}(t)\), ensuring \(T^{(k)}_\text{embed} \subset T^{(k+1)}_\text{embed}\).

Bidirectional connection: - Coarse-to-Fine Cascaded Enhancement (CE): Cross-granularity attention injects coarser contextual guidance into finer embeddings: \(\hat{X}^{(k)} = X^{(k)} \odot \mathcal{A}(X^{(k)}, \hat{X}^{(k-1)})\), where \(\mathcal{A}\) is a standard cross-attention with learned projections \(W_{q,k,v}\). - Fine-to-Coarse Hierarchical Supervision (HS): Finest-level similarity \(S^{(K)}\) (detached) acts as a teacher; all coarser levels are trained via KL-divergence distillation \(\mathcal{L}_\text{guide} = \frac{1}{K-1}\sum_{k=1}^{K-1} D_\text{KL}(\text{softmax}(S^{(k)}/\tau_d) \| \text{softmax}(S^{(K)}/\tau_d))\).

The overall objective averages contrastive alignment losses across all \(K\) levels plus \(\mathcal{L}_\text{guide}\). At inference, granularity level is selectable for performance–efficiency trade-offs; finest-grained features are the default.

Key Contributions¶

UPrompt framework: First U-Net-inspired multi-granularity prompt learning architecture for VLMs, establishing structured semantic hierarchies in both visual (spatial pooling) and textual (LLM-generated enrichment) modalities.
Bidirectional connection mechanism: CE (coarse→fine, forward pass) and HS (fine→coarse, backward pass) together ensure both global context injection and semantic consistency across granularity levels, with formal guarantees (Propositions 3.1 and 3.2 via Pinsker's inequality).
Adaptive inference: Hierarchical design enables flexible performance–efficiency trade-offs at inference time without retraining; coarser granularities offer up to 2.1× speedup at 48% compute cost.
Breadth of evaluation: 17 benchmarks spanning cross-modal retrieval, few-shot classification, base-to-novel generalization, and out-of-distribution robustness.

Results¶

Cross-modal retrieval (MSCOCO): rSum = 474.3, outperforming MAMET by +7.3 and VPKE by +4.1.
Cross-modal retrieval (Flickr30K): rSum = 571.1, best among all compared methods.
Base-to-novel generalization (11-dataset average): HM = 82.12%, +1.06% over second-best; surpasses CoCoA-Mix by +5.09% HM and TAP by +1.08% HM.
Few-shot classification (16-shot, 11-dataset average): 85.13%, surpassing GalLoP (84.50%), ProVP-Ref (83.07%), and MMA (82.76%).
OOD generalization (average across ImageNet-V2/-S/-A/-R): 61.22%, best among compared methods including MMRL (60.58%) and PSRC (60.65%).
Efficiency: Adaptive UPrompt matches fine-grained accuracy while reducing compute to 48% (2.1× speedup on ImageNet).
PSRC parity at 1/3 cost in medium-grained configuration.

Limitations¶

Textual hierarchy construction relies on Llama 3-8B for LLM-generated multi-granularity captions; this is pre-computed offline but adds a dependency on an external LLM and its prompt engineering quality.
Evaluated primarily with CLIP ViT-B/16 as backbone; generalization to other VLM architectures is presented in appendices but not fully analyzed in the main text.
The paper targets retrieval and classification; extension to dense prediction (the motivating application of U-Net) is not empirically demonstrated despite being mentioned as motivation.
Textual prompts lack inherent spatial structure, so the semantic enrichment hierarchy is a design choice (LLM-generated levels) rather than a naturally emergent property, which may not always align with visual granularities.

Relevance to Vision-Language Models¶

UPrompt directly advances the parameter-efficient adaptation of CLIP-style VLMs by replacing single-scale prompt tokens with a structured multi-granularity hierarchy, addressing a fundamental bottleneck that limits cross-task generalization. The bidirectional connection design — drawing from U-Net's skip connections and knowledge distillation — offers a principled alternative to independent multi-scale fusion used in prior works (TAP, HiCroPL, GalLoP), and the formal propositions provide theoretical grounding uncommon in prompt-learning literature. For researchers tracking VLMs, the key insight is that how granularities interact (bidirectional vs. independent fusion) matters more than how many granularities are used, with quantifiable gains across retrieval and recognition tasks. The adaptive inference mode further makes UPrompt practically relevant for deployment-constrained settings.