MQAdapter: Multi-Modal Quantum Adapter for Coarse-to-Fine VLM Fine-tuning

🕒 Published (v1): 2026-07-14 06:46 UTC · Source: Arxiv · link

Why this paper was selected

Quantum-inspired adapter for coarse-to-fine VLM fine-tuning; novel PEFT direction

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MQAdapter is a parameter-efficient adapter for few-shot VLM fine-tuning that decomposes adaptation into a coarse stage (Top-K semantic anchor retrieval via frozen CLIP) and a fine stage (cross-modal interaction via a shared Variational Quantum Circuit in a Hilbert space). By encoding visual and textual features into quantum states and exploiting superposition and entanglement, it produces more discriminative visual representations than classical Euclidean adapters. It adds only 0.078M trainable parameters and integrates as a plug-and-play module into existing PEFT frameworks.

Problem

Pre-trained VLMs (e.g., CLIP) achieve high Top-K accuracy but poor Top-1 accuracy on few-shot tasks, revealing that their learned visual representations are coarse and lack fine-grained discrimination among visually similar categories. Existing adapter-based PEFT methods focus on global visual-text alignment in classical Euclidean feature spaces and do not explicitly exploit inter-category semantic relationships to guide fine-grained visual representation refinement.

Method

MQAdapter follows a two-stage pipeline built on frozen CLIP (ViT-B/16 backbone):

Coarse stage: For input image \(I\) and \(C\) class text descriptors, cosine similarities are computed and the Top-\(K\) (\(K=5\)) highest-scoring category features \(P = [p_1, \ldots, p_K]\) are selected as semantic anchors.

Fine stage (cross-modal quantum learning): At each targeted visual layer \(\ell\) (layers 6–\(L\)), the class token \(c^\ell\) and semantic anchors \(P\) are projected to an \(N\)-dimensional space via modality-specific linear projections \(W^\ell_v\), \(W^t\). Both are then encoded by a shared 8-qubit VQC \(f_\phi(\cdot)\) consisting of: 1. Angle encoding: \(|\psi(x)\rangle = \bigotimes_{n=1}^N R_y(x_n)|0\rangle_n\) 2. Parameterized circuit: \(U(\phi) = \bigotimes_n R_y(\phi_n^{(2)}) \cdot \prod_{(n,m)\in E} \text{CRZ}(\phi_{n,m}) \cdot \bigotimes_n R_y(\phi_n^{(1)})\) with ring nearest-neighbor entanglement \(E\) 3. Measurement: Pauli-\(Z\) expectation values yield quantum-enhanced features \(f_\phi(x) \in \mathbb{R}^N\)

Cross-modal attention is computed in the quantum space: $\(\alpha^\ell = \text{softmax}\!\left(\frac{\tilde{P}^\top \tilde{c}^\ell}{\sqrt{N}}\right), \quad h^\ell = \tilde{P}\alpha^\ell\)$

The aggregated multi-modal feature \(h^\ell\) is fed again through the shared VQC, projected back to \(d_v\), and injected via residual: $\(\hat{c}^\ell = c^\ell + \gamma W^\ell f_\phi(h^\ell)\)$

The adapter is implemented with TorchQuantum and trained with temperature-scaled cross-entropy loss.

Key Contributions

• First work applying multi-modal quantum computation to parameter-efficient few-shot VLM adaptation.
• Cross-modal quantum learning mechanism: a shared VQC maps visual and textual features into a unified Hilbert space, exploiting superposition and entanglement for higher-order cross-modal interactions.
• Plug-and-play design requiring only 0.078M additional trainable parameters; integrates with MaPLe, PromptSRC, and MMRL++ without modifying their core training protocols.
• Coarse-to-fine framing that explicitly converts CLIP's strong Top-K capability into a semantic anchor signal for fine-grained discriminative refinement.

Results

Base-to-new generalization (16-shot, harmonic mean H of base/new accuracy, averaged over 11 datasets): - MaPLe → MaPLe+MQAdapter: 78.03% → 80.31% (+2.28%) - PromptSRC → PromptSRC+MQAdapter: 79.67% → 80.73% (+1.07%) - MMRL++ → MMRL+++MQAdapter: 81.54% → 82.36% (+0.82%) - MQAdapter-enhanced methods achieve the best average H among all evaluated plug-and-play approaches (including PromptSRC+DePT, TCP, PromptSRC+DPC, PromptSRC+ATP)

Few-shot image classification (averaged over 11 datasets): - MaPLe+MQAdapter improves average accuracy by +3.75% (1-shot) and +2.91% (2-shot) over base MaPLe (paper text truncated beyond this point)

Notable per-dataset gains with MaPLe: EuroSAT, DTD, UCF101, FGVC-Aircraft.

Limitations

• The VQC is simulated on classical hardware (TorchQuantum); no actual quantum speedup is demonstrated, and the quantum advantage is empirical rather than complexity-theoretic.
• The 8-qubit circuit constrains the quantum representation dimensionality and may limit expressiveness on high-dimensional feature spaces.
• Top-\(K\) anchor retrieval is a heuristic; the paper does not analyze sensitivity to \(K\) or behavior when ground-truth class falls outside the Top-\(K\) candidates.
• Evaluation is limited to CLIP ViT-B/16; generalizability to larger VLMs or non-contrastive architectures is not demonstrated.
• Paper text is truncated; full few-shot and cross-dataset/domain generalization numbers could not be fully verified.

Relevance to Vision-Language Models

MQAdapter directly addresses a core challenge in VLM PEFT: adapting CLIP-style models to fine-grained few-shot tasks without full fine-tuning. It introduces quantum circuits as an alternative to classical MLP or graph-based adapters for modeling cross-modal feature interactions, expanding the design space of VLM adapters beyond Euclidean methods. The coarse-to-fine framing—using the model's own Top-K predictions as semantic anchors—is a lightweight, model-agnostic strategy applicable to any contrastive VLM. For researchers tracking VLMs, this paper represents an intersection with quantum ML and demonstrates that high-order cross-modal interactions in a shared Hilbert space can consistently improve generalization across diverse image classification benchmarks.