Skip to content

QuARI: Query Adaptive Retrieval Improvement

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

QuARI (Query Adaptive Retrieval Improvement) is a hypernetwork-based framework that predicts a query-specific linear transformation of VLM embedding spaces, enabling fine-grained instance retrieval over massive image collections. Unlike static domain adaptation or expensive re-ranking with local descriptors or VLMs, QuARI applies a learned low-rank projection per query at negligible inference cost. It consistently outperforms both static task adaptation and large VLM re-rankers (including GPT-4o) on large-scale benchmarks.

Problem

Contrastively pretrained VLMs like CLIP and SigLIP produce global embeddings effective for coarse retrieval but insufficient for fine-grained instance-level discrimination. Existing remedies — static domain adaptation (a single learned projection) and re-ranking methods using local feature descriptors or VLMs — either fail to capture query-specific nuance or impose prohibitive computational costs at scale (millions of gallery images).

Method

QuARI trains a transformer-based hypernetwork \(H_\theta\) that takes a query embedding \(q \in \mathbb{R}^E\) and outputs (1) a refined query embedding \(q'\) and (2) a low-rank transformation matrix \(T \in \mathbb{R}^{E \times E}\) (rank \(r=64\)) parameterized as \(T = \sum_{j=1}^r u_j v_j^\top\). The hypernetwork iteratively refines \(2r\) U/V tokens over \(L\) steps using a shared transformer encoder conditioned on the query via an MLP-encoded control token with sinusoidal positional encodings. At inference, the same \(T\) linearly transforms all precomputed gallery embeddings, so retrieval requires only one forward pass through \(H_\theta\) per query. Training uses a symmetric contrastive loss with two key regularizations: (a) uniform noise added to query embeddings to bridge the text-image modality gap and enable image queries; (b) semi-positive mining — the top-2 of 100 nearest neighbors per target are treated as soft positives with KL-target logits, preventing overfitting to exact query-target pairs.

Key Contributions

  • Identifies failure modes of global VLM embeddings for instance retrieval and shows that static task adaptation is substantially inferior to query-specific adaptation.
  • Introduces the QuARI hypernetwork: a transformer predicting per-query low-rank linear projections applicable to precomputed embedding databases with minimal overhead.
  • Semi-positive sample mining and query-noise regularization as key training ingredients, each validated via ablation.
  • Demonstrates competitive or superior performance to GPT-4o on text-to-image re-ranking at orders-of-magnitude lower inference cost.

Results

  • ILIAS (mAP@1k), I2I @ 100M: SigLIP2 ViT-L baseline 20.8 → QuARI 36.2 (+15.4); SigLIP ViT-L baseline 19.6 → 30.9 (+11.3); static task adaptation (TA) for SigLIP2 ViT-L: 31.3 → QuARI 50.4 (+19.1).
  • ILIAS T2I @ 100M: SigLIP2 ViT-L baseline 19.8 → QuARI 38.7 (+18.9).
  • INQUIRE (mAP@50): SigLIP SoViT-400m baseline 34.2 → QuARI 45.4 (+11.2); SigLIP2+QuARI achieves 45.7 vs. GPT-4o 43.7 at >100× less compute.
  • ILIAS I2I re-ranking: QuARI 29.1 mAP@1k vs. AMES 26.4 (best local-feature baseline), with >100× speed advantage.
  • COCO / Flickr30k / FORB / TextCaps: SigLIP2+QuARI improves T2I R@1 from 55.2 → 77.4 (COCO) and 85.3 → 92.9 (Flickr30k) vs. base SigLIP2.
  • Ablation: removing noise regularization causes the largest drop (−15.3 mAP@1k I2I on ILIAS 100M); removing iterative refinement costs −6.8; removing semi-positives costs −5.0.

Limitations

  • The transformation is strictly linear; if the backbone lacks features relevant to a query, the projection cannot recover them.
  • Like all re-ranking methods, QuARI cannot retrieve relevant items absent from the initial top-k candidate set.
  • Rank and batch size are coupled on a single GPU (H100 80GB), limiting hyperparameter exploration at higher ranks.
  • Training requires text-image paired data; BioTrove needed VLM-generated captions (Qwen2.5-VL-7B), introducing a dependency on caption quality.

Relevance to Vision-Language Models

QuARI directly addresses a well-known weakness of CLIP-style models — their global embeddings are poorly suited for fine-grained or instance-level retrieval — by learning to reshape the VLM's embedding space conditioned on each query rather than the encoder itself. This connects to a growing line of work (RegionCLIP, FILIP, SPARC) seeking to extract more localized semantics from VLMs, but QuARI's approach is post-hoc and backbone-agnostic, making it immediately applicable to any frozen CLIP/SigLIP/SigLIP2 model. For researchers tracking VLMs, it highlights that substantial retrieval gains are available through lightweight query-conditioned adaptors rather than expensive encoder redesign, and it establishes a new strong baseline for text-to-image and image-to-image retrieval that surpasses large-scale VLM re-rankers.