Skip to content

MM-EMBED: UNIVERSAL MULTIMODAL RETRIEVAL WITH MULTIMODAL LLMS

🕒 Published (v1): 2025-01-01 · Source: ICLR · Venue: ICLR 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MM-Embed adapts a multimodal LLM (LLaVA-Next with Mistral 7B) into a bi-encoder retriever capable of handling arbitrary combinations of text, image, and interleaved text-image queries and documents across 16 diverse retrieval tasks. It introduces modality-aware hard negative mining to correct an inherent modality bias in MLLM retrievers, plus a continuous text-to-text fine-tuning stage to prevent degradation on pure-text tasks. The final model achieves state-of-the-art on the M-BEIR universal multimodal retrieval benchmark while ranking top-5 on MTEB text retrieval.

Problem

Existing retrievers are narrow in two ways: LLM-based dense retrievers are text-only, while CLIP/BLIP-based multimodal retrievers target specific cross-modal tasks (e.g., image-caption matching) and assume homogeneous document formats. No prior model handles the full universal multimodal retrieval setting—heterogeneous query/document modalities combined with diverse retrieval intents—without sacrificing competitive text-to-text retrieval. A secondary problem is that MLLMs exhibit a modality bias when fine-tuned as retrievers: given a text query with an instruction to retrieve an image, the model preferentially returns text-only documents instead.

Method

Architecture: LLaVA-Next (CLIP image encoder + Mistral 7B LLM + MLP projector) is fine-tuned as a bi-encoder. The last-token embedding of the LLM (after a one-word summarization prompt, i.e., the LLaVa-P variant) encodes each query/document into a normalized vector; retrieval is k-NN over dot products. Contrastive loss with temperature scaling is used throughout.

Stage 1 – Random negative fine-tuning (M_rand): Fine-tune on M-BEIR 1.1M training queries using in-batch random negatives; LoRA (r=8, α=64) applied to the LLM, projector fully trained.

Stage 2 – Modality-aware hard negative mining (M_hard): M_rand retrieves top-50 candidates per query. Two hard negative types are mined: (C1) candidates that rank above the positive but have the wrong modality (e.g., returns text when image is desired); (C2) candidates with the correct modality but ranked below position k′=45 (semantically too easy). During training, one negative is sampled uniformly from C1 ∪ C2 per query. This forces the model to distinguish both semantic relevance and modality compliance simultaneously.

Stage 3 – Continuous text-to-text fine-tuning (MM-Embed): M_hard(NVEmb) is further fine-tuned for 4.5K steps by uniformly mixing M-BEIR training data with 11 public text retrieval datasets (MS MARCO, HotpotQA, NQ, PAQ, etc.). Hard negatives for text datasets are mined using NV-Embed-v1.

Zero-shot reranking: A separate frozen LLaVA-Next (no fine-tuning) reranks the top-10 retrieved candidates by prompting it with task-specific True/False questions (e.g., "Does the above image match the caption? True or False"). The softmax probability of the "True" token is used as a relevance score.

Key Contributions

  • First MLLM-based universal multimodal retriever that jointly handles text, image, and interleaved text-image queries and documents across 16 heterogeneous retrieval tasks.
  • Identification and empirical characterization of modality bias in MLLM retrievers (text preference over images in cross-modal tasks).
  • Modality-aware hard negative mining: two-type hard negative construction separating wrong-modality negatives from same-modality easy negatives.
  • Continuous text-to-text fine-tuning strategy that recovers and exceeds MLLM text retrieval capability lost during multimodal fine-tuning, outperforming the base NV-Embed-v1 on MTEB.
  • Zero-shot MLLM reranking pipeline for multimodal candidates using True/False prompting, effective specifically for interleaved text-image queries.

Results

  • M-BEIR overall (R@5): MM-Embed 52.7 vs. M_hard(NVEmb) 52.3, CLIP_SF M_rand 47.4, BLIP_FF 45.5.
  • M-BEIR multi-modal query tasks: MM-Embed 47.0 vs. M_hard(NVEmb) 45.5, CLIP_SF M_hard 39.5.
  • MTEB text retrieval (nDCG@10): MM-Embed 60.3 (ranked top-5 on leaderboard) vs. NV-Embed-v1 59.36, M_hard(NVEmb) 49.7.
  • Modality-aware hard negatives improvement: MLLM retrievers gain ~5 points on M-BEIR overall; modality accuracy on MSCOCO text-to-image task jumps from 42.1% (NVEmb M_rand) to 100% (NVEmb M_hard) at R@1.
  • CIRCO composed image retrieval (mAP@5): MM-Embed retrieval 32.3 → 39.9 with reranking; +7 points over MagicLens (24.9 → 32.4) and +8.9 over E5-V (19.1 → 31.0).
  • Zero-shot reranking on VQA tasks (M-BEIR): OVEN task 8 improves from 64.1 to 65.9; InfoSeek task 8 from 57.7 to 57.3 (marginal); FashionIQ degrades (25.7 → 19.0) due to single positive label annotation artifacts.
  • Continuous vs. joint fine-tuning: Sequential (M_hard then text FT) yields 55.6 M-BEIR / 63.1 BEIR vs. joint fine-tuning 52.2 / 63.0—confirming curriculum ordering matters.

Limitations

  • Only text and image modalities are considered; audio and video are explicitly deferred to future work.
  • Zero-shot reranking degrades accuracy on single-modal query tasks (tasks 1–5 in M-BEIR), even when scaling from 7B to 34B LLaVA-Next, indicating lack of domain-specific knowledge in zero-shot prompts.
  • Continuous text fine-tuning still incurs a small multimodal retrieval drop (~0.8 points) relative to M_hard alone.
  • MLLM retrievers are substantially slower than CLIP at inference; efficiency trade-offs are only discussed in an appendix without a proposed mitigation.
  • Composed image retrieval benchmarks with sparse positive labels (FashionIQ, CIRR with one positive/query) may under-report true reranking gains due to annotation incompleteness.
  • Modality bias characterization and mitigation are studied only for text↔image cross-modal tasks; generalization to more complex interleaved-to-interleaved scenarios is not analyzed.

Relevance to Vision-Language Models

MM-Embed demonstrates a concrete pathway for repurposing generative MLLMs (LLaVA-Next) as discriminative bi-encoder retrievers, surfacing a training-time modality bias that is qualitatively distinct from VLM issues studied in generation (hallucination, sycophancy) and requiring its own remedy. The modality-aware hard negative mining technique is broadly applicable to any VLM fine-tuned with contrastive objectives over heterogeneous corpora. The zero-shot reranking result—where a frozen VLM improves over a fine-tuned retriever on compositional text-image queries—adds empirical evidence that MLLMs' cross-modal reasoning capability transfers to IR without task-specific training, relevant to the growing body of work using VLMs as judges and evaluators.