Skip to content

WAVE: Learning Unified & Versatile Audio-Visual Embeddings with Multimodal LLM

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

WAVE: unified audio-visual VLM embeddings outperform dedicated audio-visual pretraining

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

WAVE is the first MLLM-based embedding model that creates a unified representation space across text, audio, silent video, and synchronized audio-visual inputs. It introduces hierarchical feature fusion over multiple LLM layers and joint multi-task contrastive training to enable any-to-any cross-modal retrieval and prompt-aware embeddings. WAVE achieves state-of-the-art on the MMEB-v2 video benchmark and outperforms its Qwen2.5-Omni base model on audio QA.

Problem

Existing MLLM-based embedding models focus almost exclusively on static vision (image–text), leaving audio and synchronized audio-visual streams underexplored. Separate-encoder contrastive models (CLAP, AudioCLIP) lack instruction-following capability and cannot produce task-conditioned embeddings for complex downstream tasks like multimodal QA. No prior work unifies text, audio, silent video, and audio-visual streams in a single LLM-derived embedding space.

Method

WAVE is built on Qwen2.5-Omni-7B with three modality branches: a pretrained visual encoder for video frames, a speech encoder (from Qwen2.5-Omni), and a BEATs audio encoder (for environmental sound events) connected via a trainable two-layer MLP aligner. Audio tokens from the two branches are interleaved one-to-one; audio-visual tokens are frame-segment-interleaved and aligned using Time-Aligned Multimodal RoPE (TMRoPE).

Hierarchical feature fusion: For non-text inputs, the last output token from each LLM layer \(i \in \{1,\ldots,N\}\) is extracted and concatenated; a two-layer MLP with GELU activation compresses this multi-layer stack into a single embedding. For text-only inputs, standard last-token pooling on the final layer is retained.

Joint training uses two complementary objectives over ~4.9M samples: - Retrieval: symmetric InfoNCE loss over in-batch negatives for any-to-any modality pairs: $\(L_\text{Retrieval} = \frac{1}{2N}\sum_{i=1}^N (L_s^i + L_t^i), \quad L_s^i = -\log\frac{\exp(\text{sim}(e_s^i, e_t^i)/\tau)}{\sum_j \exp(\text{sim}(e_s^i, e_t^j)/\tau)}\)$ - QA: cross-entropy over a pool of \(n\) distractor answers to train prompt-aware, question-conditioned embeddings.

LoRA (rank 128, \(\alpha=2\)) is applied to the LLM backbone; only the visual aligner and LoRA modules are trained end-to-end.

Key Contributions

  • First unified audio-visual embedding MLLM supporting text, audio, silent video, and synchronized audio-visual inputs in a single semantic space.
  • Hierarchical feature fusion aggregating last tokens across all LLM layers, capturing both perceptual and semantic representations.
  • Dual-encoder audio branch (speech encoder + BEATs event encoder) with temporal interleaving via TMRoPE for precise audio-visual alignment.
  • Joint multi-modal, multi-task training (retrieval + QA) enabling prompt-aware, instruction-conditioned embeddings.
  • New benchmark suite for versatile audio-visual embedding evaluation covering audio retrieval, video-to-audio retrieval, video-to-music retrieval, and audio QA.

Results

  • MMEB-v2 Video (overall): WAVE 59.9 vs. Seed-1.6-Embedding 55.3, CAFe 42.4, GME 38.4, LamRA 35.0.
  • MMEB-v2 Video QA (with separate question prompts): WAVE 72.5 avg vs. Seed-1.6-Embedding 60.9 (~12% gap); on Video-MME: WAVE 63.4 vs. Seed 54.0.
  • Audio retrieval (R@1): AudioCaps 44.2 vs. prior SOTA 42.2; Clotho 25.6 vs. 21.5.
  • Video-to-audio retrieval (R@1): VGGSound 25.0 vs. encoder-only baseline 10.3; MusicCaps 20.4 vs. 8.6 (out-of-domain).
  • Audio QA: MMAU 76.6 and MMAR 68.1, surpassing the Qwen2.5-Omni base (71.5 / 56.7) despite no explicit audio QA fine-tuning.
  • LoVR text-to-clip R@1: 62.9 (ties GME); theme-to-clip R@1: 66.0 vs. GME 43.9.
  • Using a generic "describe the video" prompt instead of task-specific questions drops QA average from 72.5 to 51.8, confirming prompt-awareness is critical.

Limitations

  • Training requires massive compute (192 H20 GPUs, ~36 hours) and ~4.9M curated samples, limiting reproducibility.
  • Prompt-aware QA performance degrades sharply under a generic prompt (51.8 vs. 72.5), meaning retrieval and QA use-cases require different inference strategies.
  • Evaluation of audio-visual retrieval uses a self-constructed encoder-only baseline rather than published competing systems, making comparison less rigorous.
  • LoRA fine-tuning of a 7B backbone with the visual encoder frozen may cap the ceiling for purely audio-visual tasks where the base model has weaker priors.
  • The paper's ablation section is truncated in the provided text, so the contribution of individual components (hierarchical fusion, dual audio encoder, joint training) is only partially verifiable.

Relevance to Vision-Language Models

WAVE extends the MLLM-as-universal-embedder paradigm beyond static image-text (VLM2Vec, E5-V, CAFe) into the temporal, audio-visual domain, directly relevant to researchers tracking how VLMs scale to richer multimodal signals. The hierarchical feature fusion and TMRoPE alignment are practical architectural techniques that could transfer to video-VLMs more broadly. The demonstration that prompt-aware embeddings substantially outperform task-agnostic ones for multimodal QA (~12% on MMEB-v2) has implications for how VLM-based retrievers should be trained and queried. WAVE also establishes a new competitive ceiling on MMEB-v2 video, making it an important baseline for future work in instruction-following multimodal retrieval.