Skip to content

Fusion Embedding: A Unified Embedding Space for Text, Image, Video, and Audio

🕒 Published (v1): 2026-07-21 03:25 UTC · Source: Arxiv · link

Why this paper was selected

Unified text+image+video+audio embedding; fills critical audio gap in VLM retrieval stacks

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Fusion Embedding extends a frozen vision-language embedding model (Qwen3-VL-Embedding-2B) to audio by training only a small connector (16.4M params, generation 1) and modality-gated per-layer adapters (44.2M params, generation 2), producing a single vector space covering text, image, video, and audio. Non-audio outputs remain bitwise identical to the base model by construction. Audio-to-image retrieval emerges with zero paired audio-visual training data.

Problem

Embedding models built on vision-language backbones lead text/image/video retrieval benchmarks but have no audio support, while audio-text specialist systems (e.g., OEA, AuroLA) modify their backbones internally, invalidating previously computed embeddings and serving only one modality pair. No prior system adds audio to an already-deployed vision-language retriever without modifying the base model's behavior for existing modalities.

Method

Both generations freeze Qwen3-VL-Embedding-2B (the base) and the Qwen2.5-Omni audio tower (Whisper-large-v3-derived). A FusionResampler (Flamingo-style perceiver resampler, bottleneck width \(d_r = 384\), \(N = 64\) latent queries, \(L = 6\) blocks) maps audio-tower frames (dim 3584) to 64 tokens in the base's input embedding space at reserved placeholder positions; the frozen LLM's ordinary forward pass then produces the embedding via last-token pooling. Generation 2 adds bottleneck adapters (LayerNorm → \(2048 \times 384\) → SiLU → \(384 \times 2048\)) to each of the 28 decoder layers, hard-gated so the adapter branch executes only when audio tokens are present; all other inputs skip the branch entirely, preserving bitwise output identity. Training uses symmetric InfoNCE computed at every Matryoshka rung \(D \in \{2048, 1536, 1024, 512, 256, 128, 64\}\) plus a CORAL covariance penalty (\(\lambda = 0.05\)), a full-corpus frozen-text negative bank (up to 518K cached caption embeddings, zero staleness because the text tower is frozen), soft-label relaxation (\(\beta = 0.3\)) and false-negative masking (\(\tau = 0.98\)) at corpus scale \(\geq 500\)K. A two-stage recipe (pretraining then 400-step in-domain AudioCaps fine-tune) mirrors the CLAP playbook under the frozen-base constraint.

Key Contributions

  • Frozen-base audio fusion recipe: 16.4M connector (generation 1) + 44.2M modality-gated deep adapters (generation 2) extend a deployed vision-language retriever to audio with zero base parameter updates; a regression guard verifies base_drift = 0 after every run.
  • Hard-gated bitwise invariance: the generation-2 adapter gate makes non-audio outputs computation-graph-identical (not just approximately equal) to the released base, so all previously indexed text/image/video vectors remain valid without re-embedding.
  • Emergent audio-image retrieval: audio-to-image R@10 of 0.418 (29× chance level) on VGGSound-696 with zero audio-visual training pairs, via text as alignment pivot.
  • +14.5 R@10 protocol finding: matching training targets to the frozen LLM's native chat-template input format (vs. bare sequences) is the single largest improvement in the work; violating this silently costs double-digit recall.
  • Controlled negative results: LLM-rewritten captions reduce retrieval despite improved fluency; swapping to a leaderboard-superior audio tower (Dasheng) loses 16 R@10 points inside a frozen-LLM splice; wider connectors overfit; these findings map the design space for frozen-backbone audio fusion.
  • Open release: weights for both generations, Apache-2.0 code, evaluation harness, and per-number result records.

Results

  • AudioCaps audio-to-text R@10: 0.741 (gen1, v0.3), 0.775 (gen2) — frontier of the both-towers-frozen class; next competitor jina-embeddings-v5-omni: 0.672
  • AudioCaps text-to-audio R@10: 0.746 (gen1, v0.3)
  • Clotho audio-to-text R@10: 0.433 (gen1, strictly zero-shot, no Clotho training data)
  • Clotho text-to-audio R@10: 0.460 (gen1)
  • VGGSound-696 audio-to-image R@10: 0.418 (gen1), 0.443 (gen2 best checkpoint, different readout template) with zero audio-visual supervision; 29× chance level
  • Upper-bound reference: ONE-PEACE (4B params, all trained, in-domain fine-tuned): AudioCaps A→T R@10 ≈ 0.92; CLAP family (both towers trained): R@10 0.906–0.928
  • Native-protocol effect: same gen1 checkpoint scores 0.370 (bare format) vs. 0.626 (native format) on AudioCaps A→T R@10 — a 25.6-point swing from formatting alone
  • Whitening contribution: +3.3 R@10 under native protocol; +11.5 under bare protocol
  • Training cost: generation 1 — 5.1 hours on a single H100 (peak 49.5 GB); generation 2 pretrain comparable

Limitations

  • Audio performance lags specialist systems (OEA, AuroLA) that adapt the backbone internally; the frozen-base design trades retrieval headroom for deployment invariance.
  • AudioCaps A→T R@1 (0.332, gen1) remains well below full-tower CLAP systems (R@1 ≈ 0.69–0.82 range for comparable metrics in related work) and AuroLA (0.656).
  • Corpus scale is modest (518K pairs) relative to AuroLA's 1.4M-pair curated set; the paper's own scaling analysis suggests large gains remain in that data range.
  • Emergent audio-image retrieval at R@10 0.418 is well above chance but weak in absolute terms; it has not been validated against any supervised audio-visual baseline.
  • Bitwise-identity guarantee holds only under matched hardware and precision; the paper explicitly does not claim bit equality across arbitrary hardware where floating-point kernels differ.
  • Evaluation is limited to AudioCaps, Clotho, ESC-50, and VGGSound; broader audio diversity (speech, music retrieval) is not assessed.

Relevance to Vision-Language Models

This work directly exploits a vision-language embedding model (Qwen3-VL-Embedding-2B) as a frozen pivot for multimodal unification, demonstrating that a VLM's joint text-image-video representation space can be extended to new modalities with minimal additional parameters and without degrading its existing capabilities — a deployment-friendly property absent from most multimodal LLM approaches. The finding that audio-image retrieval emerges from text-only audio-caption alignment mirrors ImageBind's cross-modal emergence result, but within an already-deployed VLM rather than through modality-to-modality training, strengthening the case that strong VLM embedding spaces act as universal alignment hubs. The +14.5 R@10 protocol sensitivity result is directly relevant to anyone benchmarking or fine-tuning VLM-backbone retrievers, since using the wrong input format silently destroys recall. The modality-gated adapter design also offers a blueprint for adding future modalities to vision-language retrievers without invalidating existing indexes.