Mixture of Probes: Learning from Privileged Modalities in Multimodal LLMs Through Probing

🕒 Published (v1): 2026-07-09 18:00 UTC · Source: Arxiv · link

Why this paper was selected

Learning from privileged modalities via probing; addresses inference-time modality mismatch; practical for deployment

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

MoP (Mixture of Probes) addresses the privileged modality setting in MLLMs, where auxiliary modalities (e.g., depth, audio) are available at training time but absent at inference. It inserts learnable modality-specific and modality-general probe tokens at every layer of a shared universal encoder to disentangle and transfer cross-modal knowledge, achieving up to 65% relative improvement over strong MLLM baselines at single-modality inference.

Problem

Standard MLLMs assume training and inference modalities are identical, and treat modalities as interchangeable aligned inputs rather than complementary supervision sources. Two specific failures arise in the privileged modality setting: (i) final-layer-only adaptation of the shared encoder discards useful low-level modality-specific intermediate features; (ii) no explicit mechanism separates modality-general from modality-specific information, preventing structured cross-modal knowledge transfer.

Method

MoP augments a frozen universal modality encoder (SigLIP) with two families of learnable probe tokens that interact via layer-wise cross-attention with encoder hidden states \(H_l^m\) at every layer \(l = 1, \ldots, L\):

• Modality-specific probes \(P_s^m \in \mathbb{R}^{K_s \times d_v}\): one set per modality, updated only when that modality is active.
• Modality-general probes \(P_g \in \mathbb{R}^{K_g \times d_v}\): shared across modalities, accumulate cross-modal structure through interleaved training.

Update rule per layer: $\(P_{s,l}^m = \text{CrossAttn}_s^{m,l}(P_{s,l-1}^m, H_l^m), \quad P_{g,l} = \text{CrossAttn}_g^l(P_{g,l-1}, H_l^m)\)$

Final probe representations are projected via a shared probe connector \(g_p(\cdot)\) and concatenated with standard encoder tokens before the LLM.

MoP-X training adds two components: (1) a probe disentanglement loss penalizing off-diagonal cosine similarity between all probe pairs at every layer, $\(\mathcal{L}_{\text{dis}} = \frac{1}{LK(K-1)} \sum_{l=1}^{L} \| D \odot S_l^m \|_1\)$ to prevent probe collapse; (2) modality-interleaved batching, sampling multiple modalities within each mini-batch so shared parameters receive heterogeneous signals per optimization step. The LLM (Qwen 2.5) is fine-tuned with LoRA rank 16; the universal encoder stays frozen.

Key Contributions

• Formal definition of the privileged modality setting for MLLMs and a demonstration that naive multimodal training fails to exploit it.
• MoP architecture: structured probing across encoder depth with explicitly disentangled modality-specific and modality-general probe families.
• MoP-X training strategy: probe disentanglement loss + modality-interleaved batching.
• Comprehensive evaluation protocol across 2 domains, 8 tasks, 4 modalities treating each modality independently as the sole inference input.

Results

• Ego-in-Exo Perception (ADL, egocentric/exocentric/depth inference):
• MoP vs. Naive MLLM: +6.50 pp (egocentric), +7.14 pp (exocentric), +11.84 pp (depth).
• MoP outperforms the unimodal baseline trained only on the target modality in all three inference settings (e.g., 80.76% vs. 77.04% egocentric avg).
• MoP outperforms X-InstructBLIP and OneLLM across all three inference modalities; OneLLM performs worst (35.14–44.27% range), confirming naive alignment is insufficient.
• Music-AVQA (audio + video inference):
• MoP audio avg: 57.21% vs. 46.75% (unimodal) and 53.44% (Naive MLLM).
• MoP video avg: 77.58% vs. 69.31% (unimodal) and 73.87% (Naive MLLM).
• Overall claimed up to 65% relative improvement over strong MLLM baselines.

Limitations

• Cross-modal interaction between modality-general probes is indirect (through shared parameter updates across iterations, not within a single forward pass), to avoid exceeding the LLM context length — this may limit tight cross-modal coupling.
• Depth modality is derived synthetically via DepthAnythingV2 rather than from real depth sensors, which may not reflect true sensor diversity.
• Evaluation restricted to two domains; generalization to more diverse modality sets (e.g., IMU, tactile, radar) is untested.
• The paper text is truncated; ablation results on probe count, \(\lambda\), and MoP-X components are not fully visible.
• The universal encoder (SigLIP) is kept frozen; adaptation quality depends on how well a vision-pretrained encoder generalizes to audio and depth tokens.

Relevance to Vision-Language Models

MoP directly extends the VLM/MLLM paradigm to handle training-time-only modalities, a practical deployment scenario where VLMs must operate on cheap sensors (e.g., a single camera) while having been trained with richer sensor suites. The probing mechanism is architecturally compatible with unified encoder VLMs (e.g., OneLLM, BLIP-2-style systems) and provides a principled alternative to final-layer alignment that exploits intermediate representations — a direction increasingly relevant as VLMs scale to more modalities. The disentanglement of modality-specific vs. modality-general features also connects to open questions in VLM representation learning about what visual encoders actually learn at different depths.