Domain Adaptive Hashing Retrieval via VLM Assisted Pseudo-Labeling and Dual Space Adaptation¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

VPDS proposes a domain adaptive hashing retrieval framework that uses frozen CLIP to calibrate noisy pseudo-labels and performs independent domain alignment in the feature space and Hamming space separately. On three benchmarks (Office-Home, Office-31, Digits), it outperforms all prior domain adaptive hashing methods by substantial margins (+7.25% avg MAP over the second-best on Office-Home cross-domain, +17.30% avg on Digits).

Problem¶

Unsupervised domain adaptive hashing suffers from two compounding issues: (1) pseudo-labels for the unlabeled target domain are noisy, especially early in training when the adapted model is still biased toward the source domain; (2) existing methods either align only in the feature space or only in the Hamming space, or couple both spaces tightly (e.g., by concatenation or structural consistency constraints), which restricts the exploitation of each space's distinct properties—high semantic capacity in features vs. discriminative efficiency in compact binary codes.

Method¶

VPDS has two core components:

VLM-Assisted Pseudo-Labeling: CLIP (ViT-B/16, frozen) encodes target images and class-name text prompts ("a photo of [CLASS]") to produce zero-shot predictions y_clip. The final pseudo-label is ỹ = argmax(β·ŷ + (1−β)·y_clip), where β = e/E increases linearly from 0 to 1 over training. This gradually shifts reliance from VLM to the adapting model. Since CLIP is frozen, y_clip is computed once in epoch 1 and cached.

Dual Space Adaptation (decoupled): Two classifiers—C_F in the feature space and C_H in the Hamming space—are trained independently. Pseudo-labels are generated in the feature space (richer semantics) and used unidirectionally to supervise Hamming space learning (L_t). Domain alignment is performed separately in each space via prototype-based global alignment: momentum-updated memory banks store per-class prototypes for source and target in each space; alignment loss is the negative mean cosine similarity between source and target prototype matrices (L_PF, L_PH). Information maximization (L_i) is applied in both spaces to encourage confident, diverse predictions. The overall loss is L = L_s + η·L_t + γ·L_i + L_PF + L_PH.

VGG backbone is used as the feature encoder; a hash encoder maps features to Hamming space with tanh during training and sign at inference.

Key Contributions¶

Decoupled dual-space alignment: independent alignment in feature and Hamming spaces without enforcing cross-space structural consistency, enabling flexible exploitation of each space's strengths.
VLM-calibrated pseudo-labeling: linearly weighted combination of CLIP zero-shot predictions and adapted model predictions, with minimal compute overhead (single cached forward pass).
Model-agnostic pseudo-labeling module that can be plugged into other unsupervised domain adaptation frameworks.
Demonstrated scalability to fully unsupervised hashing (VLM pseudo-labels replace source ground truth) and out-of-sample (generalization) retrieval settings.

Results¶

Office-Home cross-domain (64-bit MAP): VPDS avg 75.56% vs. CPH (second-best) 68.31% — +7.25%.
Office-31 cross-domain (64-bit): VPDS achieves 87.22%, 91.40%, 65.88%, 98.91%, 64.88%, 100% on A→D, A→W, D→A, D→W, W→A, W→D tasks; avg 84.88% vs. CPH 72.96%.
Digits cross-domain (avg across all bit lengths): VPDS 85.83% vs. COUPLE (second-best) 68.53% — +17.30%.
Single-domain retrieval (Table 3): VPDS outperforms all baselines on all three sampled tasks (P→R, A→D, MNIST→USPS) at all bit lengths.
Ablation (Office-Home): Removing VLM pseudo-labeling drops avg MAP from 66.40% to 64.80% (w/ Pred) or 57.08% (w/ Prot); removing dual-space and using only Hamming drops to 39.95%.
Unsupervised setting (no source labels): Still outperforms most domain-adaptive baselines.

Limitations¶

Uses VGG as the backbone rather than stronger modern encoders (e.g., ViT, ResNet-50), which may limit the ceiling performance and comparability to newer methods.
CLIP predictions (y_clip) are cached after epoch 1, so any target domain drift during training is not reflected in the VLM guidance signal beyond epoch 1.
The linear schedule for β is fixed and not adaptive to dataset difficulty or training dynamics; the paper shows gradient vanishing for MNIST→USPS at γ ≥ 0.3.
Evaluation limited to three relatively small/standard benchmarks; scalability to large-scale retrieval (millions of samples) is not demonstrated.
No analysis of failure modes when CLIP's zero-shot accuracy is poor for domain-specific classes (e.g., specialized medical or satellite imagery).

Relevance to Vision-Language Models¶

This paper demonstrates a practical pattern for injecting VLM zero-shot knowledge into a non-VLM downstream pipeline: rather than fine-tuning CLIP or building a VLM-native retrieval system, it uses frozen CLIP as an external calibrator to bootstrap noisy pseudo-labels in the early training phase. This is directly relevant to the broader VLM literature on zero-shot generalization and domain robustness, showing that CLIP's image–text alignment can substitute for labeled supervision during domain shift. The work also touches on a recurring challenge in VLM adaptation—pseudo-label noise under distribution shift—and offers a lightweight, model-agnostic solution applicable to other unsupervised VLM-assisted frameworks.