Skip to content

ZooClaw-FashionSigLIP2: Distilled Fine-tuning for Robust Fashion Retrieval

🕒 Published (v1): 2026-06-26 04:13 UTC · Source: Arxiv · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ZooClaw-FashionSigLIP2 is a fashion-specialized SigLIP2-base model that resolves the specialization vs. OOD generalization tradeoff via full fine-tuning with knowledge distillation (LwF) followed by WiSE-FT weight interpolation at \(\alpha=0.4\). It outperforms LoRA, larger backbones up to 1B parameters, and existing fashion-specific models (Marqo-fashionCLIP, Marqo-fashionSigLIP) on every benchmark in their suite. The authors also release the ZooClaw-Fashion benchmark and a TREC-style pooled re-evaluation of Fashion200k exposing structural biases in its public ground truth.

Problem

Adapting a foundation vision-language encoder to a specialized retrieval task creates a fundamental tradeoff: gains on the target distribution come at the cost of broad OOD generalization. Fashion retrieval is a stringent instance of this problem because it requires fine-grained visual and textual distinctions (neckline, fabric texture, styling details) that general-purpose pre-training does not capture, while production systems must simultaneously serve short keyword queries and long attribute-rich descriptions across unseen public catalogs with different query styles.

Method

  • Base model: SigLIP2-base-patch16-384, full fine-tuning (not LoRA — ablations show no LoRA configuration matches full FT)
  • Generalized Contrastive Loss (GCL): Extends InfoNCE with graded relevance scores \(r_i \in [0,1]\); in-batch negatives from highly relevant pairs are downweighted via \(w_{ij} = 1 - r_j \cdot \mathbb{1}[i \neq j]\) to avoid pushing apart false negatives
  • Multi-task formulation: Two contrastive tasks — short-query and long-query retrieval — combined as \(\mathcal{L}_{\text{con}} = \lambda_s \mathcal{L}_{\text{short}} + \lambda_l \mathcal{L}_{\text{long}}\) with \(\lambda_l = 1.0\), \(\lambda_s = 0.5\)
  • Knowledge distillation (LwF): Frozen base model as teacher on image encoder; cosine distance between student and teacher embeddings: \(\mathcal{L}_{\text{LwF}} = \frac{1}{N} \sum_i (1 - \cos(f_\theta(v_i), f_{\theta_0}(v_i)))\), with \(\lambda_{\text{LwF}} = 1.0\)
  • WiSE-FT weight interpolation: \(\theta_\alpha = (1 - \alpha) \cdot \theta_0 + \alpha \cdot \theta_{\text{ft}}\); deployed at \(\alpha = 0.4\), selected by maximizing the minimum margin over both baselines across all benchmarks
  • Data: Short queries (\(\sim\)5 words, attribute-sampled + Gemma-4-31B rewrite) and long queries (\(\sim\)40 words, full attribute set → description) generated from a proprietary Gensmo catalog; VLM-assigned graded relevance scores (0–10)
  • ZooClaw-Fashion benchmark: 2K queries, 12K images, 1,355 fine-grained product groups, dual short/long queries per product

Key Contributions

  • ZooClaw-FashionSigLIP2: a fashion-specialized VLE that resolves the specialization-generalization tradeoff via distilled fine-tuning + WiSE-FT
  • Systematic demonstration that full fine-tuning consistently outperforms LoRA at all ranks/regularizations for fashion retrieval, and that adding external training data (Marqo-Fashion) counterintuitively degrades both in-domain and OOD performance
  • Release of the ZooClaw-Fashion benchmark (2K queries, 12K images, 1,355 product groups, dual-query) with open-source evaluation artifacts
  • TREC-style pooled re-evaluation of Fashion200k with 102,494 held-out Gemma-4-31B graded judgments, exposing that the original ground truth is biased toward caption-source instance recovery
  • Open-source model weights (HuggingFace: srpone/zooclaw-fashionsiglip2) and benchmark dataset

Results

  • ZooClaw-Fashion (long query): R@1=0.423, R@10=0.738, MRR=0.533 — leads all baselines
  • ZooClaw-Fashion (short query): R@1=0.449, R@10=0.795, MRR=0.567 — leads all baselines
  • Fashion200k (pooled qrels, thr≥3): R@10=0.286, MRR=0.136, nDCG@10=+0.012 over Marqo-fashionSigLIP
  • H&M: R@10=0.925, MRR=0.066 — leads all baselines
  • Outperforms Marqo-fashionCLIP, Marqo-fashionSigLIP, SigLIP2 zero-shot, and LLM2CLIP (Llama-3.1-8B encoder) on every metric
  • Full FT + LwF yields ZooClaw-Fashion R@10=0.768 vs. best LoRA + LwF at 0.749; Fashion200k R@10=0.248 vs. 0.249 (tie)
  • Backbone scaling: ZooClaw-Fashion improves with size (base→giant), but Fashion200k trends slightly downward (0.261→0.239) and H&M saturates at so400m — scaling alone insufficient for OOD

Limitations

  • Training data is proprietary and not released; only the evaluation benchmark and model weights are open-sourced
  • Fashion200k evaluation uses a new pooled qrels protocol (LLM-judged, not human), making direct comparisons with previously published numbers on the original qrels non-trivial
  • Both ZooClaw-Fashion and H&M ground truth are LLM-generated (Gemma-4-31B), not human-curated relevance judgments
  • The counterintuitive finding that adding more external data (Marqo-Fashion) always hurts both in-domain and OOD performance suggests the recipe is brittle to data distribution mismatch
  • WiSE-FT \(\alpha\) is selected on the evaluation benchmarks; while the authors argue minimal overfitting risk, it's not a truly held-out selection
  • Results are modest in absolute terms on Fashion200k (pooled R@10=0.286), indicating room for improvement on large-scale OOD retrieval

Relevance to Vision-Language Models

This paper directly addresses the central tension in VLM deployment: how to specialize a pre-trained VLE to a domain without destroying its broad generalization. It provides a practical, reproducible recipe (full FT + LwF + WiSE-FT) that beats both parameter-efficient methods (LoRA) and the naive scaling approach (larger backbones). For the VLM community, it also contributes a careful methodological critique of benchmark design — showing that the common practice of deriving ground truth from caption-source links systematically biases evaluation toward models trained on similar caption distributions.