Instance-Level Composed Image Retrieval¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper introduces i-CIR, a new evaluation benchmark for composed image retrieval (CIR) that operates at the instance level (same specific object) rather than semantic class level, paired with explicit hard negatives across three dimensions. It also proposes BASIC, a training-free method using pre-trained CLIP that achieves state-of-the-art on i-CIR and four existing CIR datasets.

Problem¶

Existing CIR benchmarks (CIRR, FashionIQ, CIRCO) are built by automatically pairing visually similar images and describing their difference—a process that yields poor-quality text queries, high false-negative rates, and text-only shortcuts where the image query is redundant. None operate at instance level (same specific object), and none include systematically mined hard negatives across visual, textual, and composed dimensions. As a result, reported progress overstates true compositional retrieval ability.

Method¶

i-CIR dataset: Human annotators define object instances (e.g., Temple of Poseidon) and meaningful text modifications. Seed images and seed sentences are used to retrieve candidate images from LAION via CLIP similarity. After automated filtering (perceptual hashing, resolution), annotators label composed positives; all remaining candidates become negatives, with a structured subset designated hard negatives (visual: same instance, wrong modification; textual: correct modification, different instance; composed: approximately satisfies both). Each instance has its own database to allow error-free labeling.

BASIC (training-free): 1. Centering: Subtract precomputed image mean µ^v (from LAION) and text mean µ^t from all features. 2. Semantic projection: Build a contrastive covariance matrix C = (1−α)C₊ − αC₋ from object-oriented (C₊) and style/context (C₋) text corpora; extract top-k=250 eigenvectors as projection matrix P ∈ ℝ^{d×k}. Project centered image features P⊤x̄^v to suppress style variation and retain object identity cues. 3. Text contextualization: Wrap raw text queries (e.g., "during sunset") with random terms from C₊ to form caption-like phrases; average their CLIP embeddings to bridge VLM pre-training distribution gap. 4. Min-based normalization: Affinely rescale dot-product scores using empirical minimum s_min so that s_min→0 and 0→1. 5. Harris fusion: s̃^f = s̃^v · s̃^t − λ(s̃^v + s̃^t)², penalizing items where only one modality is highly activated, analogous to the Harris corner criterion. Optionally apply image query expansion (weighted top-k database features).

All projection and centering operations can be folded into the query side at inference, leaving the stored CLIP database index unmodified.

Key Contributions¶

i-CIR: 202 instances, 750K images, 1,883 composed queries, with three classes of hard negatives; equivalent in difficulty to retrieval over >40M random LAION distractors.
Compositional validity analysis: on i-CIR, combining both modalities yields +14.9 mAP (+490% relative) over the best unimodal baseline; CIRR achieves only +3.0 mAP (+11%), demonstrating legacy dataset inadequacy.
BASIC: training-free CIR pipeline achieving new SOTA on i-CIR (31.64% macro-mAP) and on ImageNet-R (32.13%), NICO++ (31.65%), MiniDN (39.58%), LTLL (41.38%), outperforming all ZS-CIR methods including MagicLens and FreeDom.
Ablation characterizing each component's contribution and showing the projection matrix transfers across domains using generic corpora.

Results¶

All metrics are mAP (%). BASIC uses CLIP ViT-L/14; CompoDiff uses ViT-G/14.

i-CIR: BASIC 31.64 vs. CoVR-2 28.50 (next best), MagicLens 27.35, FreeDom 17.24
ImageNet-R: BASIC 32.13 vs. FreeDom 29.91
NICO++: BASIC 31.65 vs. CoVR-2 24.93
MiniDN: BASIC 39.58 vs. FreeDom 37.27
LTLL: BASIC 41.38 vs. FreeDom 33.24
Without query expansion (BASIC†): still SOTA on all five datasets
Ablation: centering alone lifts i-CIR from 17.48→28.33; adding semantic projection further to 33.48; full BASIC 34.35 (with QE) or 31.64 (without)
Text contextualization removal drops i-CIR from 31.64→25.85

Limitations¶

Instance-level labeling requires significant human annotation effort, limiting dataset scale compared to automatically constructed benchmarks.
BASIC's logical AND assumption degrades when one modality dominates (e.g., purely textual transformations or CIRR-style entangled queries).
Query expansion hurts on i-CIR (34.35 without QE vs. 31.64 with—correction: QE is the last row in ablation, so without QE is 33.48 and full BASIC without QE is 31.64; with QE is 32.13/31.64 depending on dataset)—specifically, query expansion decreases performance on i-CIR while helping category-level datasets.
Hard negative counts and database sizes vary widely per instance (951–10,045 hard negatives), introducing evaluation variance.
Dataset covers only 202 instances; generalization to rarer object types is untested.

Relevance to Vision-Language Models¶

This paper directly benchmarks and leverages VLM (CLIP) capabilities for composed retrieval, revealing that existing CIR datasets inadvertently reward single-modality text shortcuts, masking genuine VLM compositional understanding. The BASIC method demonstrates that carefully designed feature-space operations—centering, subspace projection guided by LLM-generated corpora, and cross-modal fusion—can extract substantially more compositional signal from a frozen CLIP backbone than prior training-free or even fine-tuned approaches. For VLM researchers, i-CIR provides a rigorous probe of instance-level visual grounding under textual modification, a capability gap distinct from standard VQA or image-text matching benchmarks. The text contextualization technique (embedding text queries in caption-like phrases sampled from an object corpus) also speaks to the distribution mismatch between VLM pre-training and downstream query formats—a broadly relevant issue in zero-shot VLM deployment.