Skip to content

MoRA: Missing Modality Low-Rank Adaptation for Visual Recognition

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

Low-rank adaptation handles missing modalities in VLM visual recognition tasks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MoRA is a parameter-efficient fine-tuning (PEFT) method for vision-language models that handles missing modalities during both training and inference. It introduces modality-specific LoRA parameters alongside shared cross-modal parameters that communicate through Gram matrices in a common rank space, enabling bidirectional knowledge transfer without inference overhead.

Problem

Pre-trained VLMs (e.g., CLIP) assume modality completeness at inference time, but real-world deployments face missing modalities due to privacy, cost, or sensor failures. Existing prompt-based methods (MMP, DCP) add learnable tokens that introduce inference latency, fail to capture full cross-modal relationships, and discard missing-modality features. Standard LoRA extensions do not model cross-modal interactions in dual-encoder architectures.

Method

MoRA decomposes each encoder's weight update into two components. For encoder \(e \in \{v, t\}\), the updated weight is:

\[W_e = \alpha_e W_0^e + B_e A_e + \alpha_s W_0^e + S_e^T G_{\bar{e}} S_e\]

where \(B_e A_e \in \mathbb{R}^{d_e \times d_e}\) are modality-specific low-rank parameters (rank \(r\)), and \(S_e \in \mathbb{R}^{r \times d_e}\) are shared parameters. Because vision (\(d_v{=}768\)) and text (\(d_t{=}512\)) encoders have different dimensions, direct shared-parameter multiplication is incompatible. MoRA resolves this via Gram matrices: \(G_v = S_v S_v^T \in \mathbb{R}^{r \times r}\) and \(G_t = S_t S_t^T \in \mathbb{R}^{r \times r}\). The cross-modal update for each encoder uses the other modality's Gram matrix (\(G_{\bar{e}}\)), operating in a dimension-agnostic rank space. Crucially, \(S_e^T G_{\bar{e}} S_e \in \mathbb{R}^{d_e \times d_e}\) can be absorbed into \(W_0^e\) before inference, incurring zero latency overhead.

Key Contributions

  • MoRA PEFT formulation with dual modality-specific + modality-shared low-rank parameters for dual-encoder VLMs.
  • Gram-matrix-based cross-modal knowledge transfer that resolves dimension mismatch between vision and text encoders without projection layers or added inference cost.
  • Demonstration that both inter-modal alignment and the modality gap carry complementary information critical for missing-modality robustness.
  • All learned parameters absorb into pre-trained weights at inference, matching frozen-backbone latency.

Results

  • Overall: MoRA improves over SOTA (DCP) by +5.30% (MM-IMDb), +1.91% (Food101), +8.51% (Hateful Memes) averaged across missing ratios of 50%/70%/90%.
  • Efficiency: Only 0.16M trainable parameters vs. DCP's 5.92M; inference time 88.58 ms/sample vs. DCP's 342.04 ms (≈25.9% of DCP's latency).
  • Trainable parameter fraction: 0.11% of full fine-tuning parameter count.
  • Extreme missing (90%, Hateful Memes): MoRA at 90% missing matches DCP at 50% missing.
  • Cross-modal retrieval generalization (Table 6): CLIP4CIR + MoRA improves Recall@1 from 43.34 → 60.50, R@5 from 76.99 → 85.00 on MS-COCO.
  • Inter-modal distance (Food101, Table 2): MoRA L2 distance 9.99° / angle 77.07° vs. CLIP original 1.18° / 72.44°, far closer than FFT (22.61° / 91.64°) or DCP (15.78° / 86.92°).
  • Ablations (Table 5): removing Gram matrices (w/o Gram) drops avg. −2.66%; removing shared parameters (w/o Specific) drops −2.78%.

Limitations

  • Evaluated only on dual-modality (vision + text) settings; generalization to \(>2\) modalities is unaddressed.
  • Gram-matrix approach implicitly assumes structural similarity across modality encoder weights in rank space; this may not hold for highly heterogeneous architectures.
  • Missing-modality simulation uses simple all-ones image / empty string substitution, which may not reflect realistic degradation patterns.
  • Results are on classification/retrieval benchmarks; applicability to generative VLMs (e.g., LLaVA) is not demonstrated.

Relevance to Vision-Language Models

MoRA directly advances PEFT for dual-encoder VLMs (CLIP-family) by addressing a practically important but underexplored failure mode: missing modalities at inference. The Gram-matrix mechanism for dimension-agnostic cross-modal parameter sharing is a novel architectural contribution that could generalize to other asymmetric dual-encoder designs. The work also provides empirical grounding for how the modality gap in CLIP's embedding space functions as complementary rather than merely noise—informing future alignment strategies. For researchers studying multimodal robustness and efficient adaptation, MoRA establishes a new efficiency-performance frontier, combining zero inference overhead with state-of-the-art missing-modality handling.