Skip to content

Multi-Modal Interactive Agent Layer for Few-Shot Universal Cross-Domain Retrieval and Beyond

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MAIL (Multi-Modal Interactive Agent Layer) introduces a parameter-efficient fine-tuning method for CLIP that couples agent layers across vision and text encoders via a bottleneck bridge function, enabling aligned cross-modal parameter updates at training time with zero added inference cost via reparameterization. It is evaluated on a newly formalized task, Few-Shot Universal Cross-Domain Retrieval (FS-UCDR), where models must generalize to unseen domains and categories from very few training samples.

Problem

Universal Cross-Domain Retrieval (UCDR) requires large, multi-domain annotated datasets, making it impractical in low-data regimes. Existing PEFT methods applied to CLIP either lack cross-modal coupling (adapter/prompt-independent variants) or incur inference overhead when coupling is introduced (MaPLe, MMA). No prior work has explored modality coupling within the third PEFT category—partially fine-tuned / reparameterizable methods—leaving a gap where inference efficiency and cross-modal alignment cannot be achieved simultaneously.

Method

MAIL freezes selected target layer pairs in CLIP's vision and text encoders and inserts trainable agent layers (AL) at up to five positions per transformer block: after LayerNorm (×2), MHSA output, MLP output, and final projection. Each AL is a lightweight scaling-and-shifting transform (AL ∘ OP(x) = OP(x) · Λ(a) + b) initialized as identity, which approximates row-wise weight updates of the frozen layer. Cross-modal coupling is achieved by a bottleneck bridge function that adds a language-to-vision projection to the vision scaling vector: ā_v = a_v + W_up · W_down · a_t (rank-8 default, LoRA-style initialization). Gradients thus flow from text ALs to vision ALs during training, enforcing aligned updates across modalities. At inference, all ALs are reparameterized into the frozen layers, restoring exact CLIP inference complexity.

Key Contributions

  • Formal definition of FS-UCDR: retrieval with queries from unseen domains and classes, trained with only k shots per class per source domain.
  • Empirical finding that modality-coupled methods (MCMs) consistently outperform modality-independent counterparts under FS-UCDR, and that text-to-vision guidance direction is superior to vision-to-text or bidirectional coupling.
  • MAIL: first MCM operating within the reparameterizable PEFT family, achieving zero additional inference overhead while enabling cross-modal gradient communication.
  • State-of-the-art on FS-UCDR, FS-UD CDR, FS-UC CDR, and competitive on 11-dataset few-shot classification benchmarks.

Results

  • FS-UCDR (2-shot, DomainNet, Average UnseenGallery mAP₂₀₀): MAIL 62.05 vs. MMA 60.31, MaPLe 60.29, IVLA 59.12; surpasses ProS (64.67) on most individual domains despite ProS using ~140× more training data.
  • FS-UD CDR (DomainNet, Average mAP₂₀₀): MAIL 63.11 vs. MMA 59.86, MaPLe 61.22; matches ProS* (63.32) with 1/140th the data.
  • FS-UC CDR (Sketchy mAP_all / TU-Berlin Prec₁₀₀): MAIL 73.46 / 75.10 vs. MaPLe 71.86 / 73.73 (best prior).
  • Base-to-novel generalization (16-shot, 11 datasets, HM average): MAIL 81.10 vs. DeKg 80.44 (prior best), without any regularization loss.
  • Cross-dataset evaluation (average): MAIL 52.80 vs. MMA 49.24, DeKg 51.37.
  • Domain generalization (average over 4 ImageNet variants): MAIL 77.80, matching MMA 77.32 and ahead of PromptSRC 77.80 (tied).
  • MAIL uses only 0.50% tunable parameters vs. LoRA's 12.84%, while achieving higher mAP.

Limitations

  • Sequential application of agent layers (AL ∘ OP(x)) increases training time and memory relative to MaPLe and MMA (magnitude not stated in main text; quantified in appendix).
  • Bridge function rank tuning is required; performance degrades slightly above rank 16 due to overfitting risk.
  • Only language-to-vision coupling is explored; vision-to-language coupling degrades performance, potentially limiting applicability to tasks where visual features are semantically richer.
  • Evaluated only on CLIP-ViT backbone; generalization to other VLM architectures not demonstrated.

Relevance to Vision-Language Models

MAIL directly addresses a core challenge in VLM adaptation: how to introduce cross-modal parameter coupling during fine-tuning without increasing inference cost, a property critical for deploying CLIP-based models at scale. The finding that text-guided vision alignment outperforms bidirectional or vision-guided alignment offers a concrete design principle for future VLM PEFT work. The reparameterization trick generalizes beyond retrieval—it applies to any task requiring efficient CLIP fine-tuning—making MAIL relevant to the broad VLM adaptation literature (prompt tuning, adapters, LoRA variants). The FS-UCDR benchmark itself is a challenging new evaluation axis that exposes cross-modal generalization limits of existing PEFT strategies.