Skip to content

FlowCIR: Semantic Transport via Flow Matching for Zero-Shot Composed Image Retrieval

🕒 Published (v1): 2026-07-02 15:02 UTC · Source: Arxiv · link

Why this paper was selected

Zero-shot composed image retrieval via flow matching; novel semantic transport approach for VLMs

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

FlowCIR reframes zero-shot composed image retrieval (ZS-CIR) as conditional semantic transport in VLM embedding space, replacing textual inversion with a lightweight flow-matching network that maps a relative-instruction embedding to a target-oriented query embedding conditioned on the reference image. An inference-only Multi-Negative Steering strategy handles negation-heavy instructions by geometrically repelling the composed representation from negated concept directions. The whole system trains in ~0.5 hours on a single RTX 3090, substantially undercutting prior methods' multi-GPU requirements.

Problem

Existing ZS-CIR methods rely on textual inversion: projecting a reference image into pseudo-text tokens and concatenating them with the modification instruction in text space. This is lossy for fine-grained visual semantics (a few tokens cannot faithfully encode a rich image) and computationally expensive (typically hours of large-batch contrastive training on multiple A100s). Additionally, CLIP-style VLMs are geometrically ambiguous under negation ("a dog" and "no dog" occupy nearby directions), so negation-heavy instructions systematically fail.

Method

Core transport. FlowCIR learns a conditional velocity field \(f_\theta(\mathbf{x}_t, t, \mathbf{x}_{I_r})\) that transports the \(\ell_2\)-normalized relative-instruction embedding \(\mathbf{x}_{T_r}\) toward the target-side text embedding \(\mathbf{x}_{T_t}\), conditioned on the frozen reference-image embedding \(\mathbf{x}_{I_r}\). Training uses the rectified-flow objective on linearly interpolated paths: $\(\mathcal{L}_\mathrm{FM} = \mathbb{E}\!\left[\left\|f_\theta(\mathbf{x}_t, t, \mathbf{x}_{I_r}) - (\mathbf{x}_{T_t} - \mathbf{x}_{T_r})\right\|_2^2\right]\)$ A contrastive retrieval loss \(\mathcal{L}_\mathrm{RET}\) (top-\(K\) hard-negative InfoNCE between the one-step-predicted target text embedding and gallery image embeddings) sharpens discrimination. The full objective is \(\mathcal{L} = \mathcal{L}_\mathrm{FM} + \lambda\mathcal{L}_\mathrm{RET}\). At inference, a single ODE step suffices: \(\hat{\mathbf{x}}_{T_t} = \mathbf{x}_{T_r} + f_\theta(\mathbf{x}_{T_r}, \mathbf{x}_{I_r}, 0)\).

Backbone. Frozen CLIP (ViT-B/32 or ViT-L/14) provides all image/text embeddings; only the flow network (a deep residual MLP from MAR) is trained. Synthetic supervision comes from HQ-Edit-200k.

Multi-Negative Steering (MNS). At inference, negation-containing instructions are parsed (rule-based keywords + TinyLlama-1.1B) to extract an affirmative intent \(\mathbf{x}_{T_r^a}\) and negated concepts \(\{\mathbf{x}_{T_{r,k}^n}\}_{k=1}^K\). Each negative defines a hyper-spherical cap; a closed-form steering direction repels the embedding from all negative caps: $\(\tilde{\mathbf{d}}_k = \mathbf{x}_{T_r^a}\frac{\sin(\alpha + \theta_k/2)}{\sin\theta_k} - \mathbf{x}_{T_{r,k}^n}\frac{\sin(\alpha - \theta_k/2)}{\sin\theta_k}\)$ The steered embedding \(\hat{\mathbf{x}}_{T_r}\) replaces \(\mathbf{x}_{T_r}\) as the flow's starting point, requiring no additional training.

Key Contributions

  • FlowCIR paradigm: formulates ZS-CIR as conditional semantic transport via flow matching, avoiding textual inversion's token-compression bottleneck.
  • Theoretical grounding: proves that unconditional flow collapses to \(\mathbb{E}[\mathbf{x}_{T_t}|\mathbf{x}_{T_r}]\) (mean collapse) when the target distribution is multi-modal, and that reference conditioning recovers exact point-to-point semantic transport.
  • Multi-Negative Steering: inference-only, training-free geometric strategy to handle negation/removal instructions in CLIP space.
  • Training efficiency: ~0.5h on one RTX 3090, versus 4–231 GPU-hours on multiple A100s for competing methods.

Results

  • CIRR (ViT-B/32): R@1 25.5, R@5 56.5, R@10 69.8; best among public-data baselines on R@5 (+0.8pp over iSEARLE).
  • CIRCO mAP@5 (ViT-B/32): 13.1, a +23.6% relative gain over iSEARLE (10.6).
  • CIRR (ViT-L/14): R@1 26.2, matching/exceeding Context-I2W (25.6) and LinCIR (25.0) at a fraction of the cost.
  • CIRCO mAP@5 (ViT-L/14): 14.9, +14.6% over LinCIR+CIG (13.0); trails MagicLens (29.6) which uses 36.7M private triplets and 128 TPUs.
  • Fashion-IQ average Recall@10/50 (ViT-L/14): 29.7 / 48.9; surpasses SEARLE, Context-I2W, and LinCIR; below MagicLens (30.7 / 52.5).
  • Training cost: 0.5h (single RTX 3090) vs. 4h–128h (8–128 A100s) for baselines.

Limitations

  • Negation-detection heuristic (keyword matching + TinyLlama) may mis-trigger on non-negation queries or miss implicit negations, introducing noise in the steered embedding.
  • Still trails MagicLens on all benchmarks; that method uses 36.7M private triplets and large TPU clusters, so the gap may reflect training data scale rather than architecture.
  • One-step inference approximates the true ODE trajectory; multi-step integration is not evaluated in the truncated text, leaving potential accuracy–latency trade-offs unexplored.
  • MNS requires LLM parsing at inference time, adding latency that is not quantified.
  • Reliance on frozen CLIP means the method inherits CLIP's known weaknesses in negation geometry rather than correcting them at the representation level.

Relevance to Vision-Language Models

FlowCIR directly exploits the shared embedding space of frozen VLMs (CLIP), treating semantic composition as a transport problem within that space rather than via language-side token manipulation — a fundamentally different way to leverage VLM geometry. The Multi-Negative Steering highlights and partially patches a structural weakness of contrastive VLMs: the geometric ambiguity of negation in CLIP text space, a documented failure mode relevant to any VLM-based retrieval or grounding system. For researchers tracking VLMs, this work demonstrates that lightweight modules trained on top of frozen VLM embeddings can rival heavily trained inversion-based approaches, underscoring the richness of pre-trained VLM feature spaces. It also connects to the broader trend of using flow/diffusion-based transport within VLM embedding spaces (cf. FMA, HFM, FlowComposer) for cross-modal tasks beyond generation.