Skip to content

Unified Vision–Language Modeling via Concept Space Alignment

🕒 Published (v1): 2026-01-01 · Source: ICLR · Venue: ICLR 2026 · link

Why this paper was selected

ICLR 2026; V-SONAR: extends multilingual text embeddings to 1500-language vision

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

V-SONAR extends the massively multilingual SONAR embedding space to image and video via post-hoc teacher-student alignment, producing a single latent space shared by 1500 text languages and visual inputs. The authors then demonstrate that LCM, a latent diffusion language model trained on text only, can zero-shot process V-SONAR visual embeddings, and fine-tune it into V-LCM, a vision-language model that outperforms comparable VLMs across 61 of 62 tested languages on M3IT.

Problem

Existing multilingual embedding spaces (SONAR, OmniSONAR) cover text and speech but exclude vision, preventing language-agnostic latent-space models like LCM from operating on image/video inputs. Standard VLMs also lack genuine multilingual generalization: models such as InternVL and Qwen-VL are competitive in high-resource languages but degrade in low-resource settings.

Method

V-SONAR construction uses teacher-student alignment: the Perception Encoder (PE) backbone is kept as the vision encoder, and a lightweight projector (temporal attention + attention pooling) is trained to minimize MSE between visual embeddings \(z_v = f_\theta(V_i)\) and frozen SONAR text embeddings \(z_t = g(T_i)\):

\[\mathcal{L}_{\text{align}} = \frac{1}{N}\sum_{i=1}^{N} \|f_\theta(V_i) - g(T_i)\|_2^2\]

Training follows a coarse-to-fine curriculum: Stage 1 — 12M image–caption pairs (coarse grounding); Stage 2 — 2M synthetic video captions (temporal adaptation); Stage 3 — 200K human-annotated video captions (fine-grained alignment). Only projector parameters and PE are updated; SONAR is frozen.

V-LCM fine-tunes LCM via vision-language instruction tuning. Visual inputs are encoded by V-SONAR, textual inputs by SONAR; embeddings are concatenated into a unified sequence and trained with LCM's latent diffusion objective for next-embedding prediction (variance-preserving schedule, two-tower denoiser architecture). No discrete visual tokens are used; generation operates entirely in the continuous latent space.

Key Contributions

  • Post-hoc coarse-to-fine alignment pipeline that maps Perception Encoder representations into the SONAR space, creating V-SONAR with no SONAR parameter updates.
  • State-of-the-art zero-shot video retrieval (R@1 = 73.03 on PE-VIDEO vs. 63.91 for PECoreG) and video captioning (BLEU 39.0 on PE-VIDEO, 23.9 on DREAM-1K) via V-SONAR + OmniSONAR decoder.
  • Zero-shot visual understanding by LCM (text-only trained), demonstrating that aligned concept spaces transfer across modalities without visual fine-tuning.
  • V-LCM, the first latent diffusion VLM, matching 1–3B VLMs on captioning/QA while outperforming all competitors in 61/62 M3IT languages.
  • Embedding space analysis metrics (Alignment Consistency, Trace, logdet) to diagnose cross-modal alignment quality.

Results

Zero-shot video retrieval (Table 2): - V-SONAR R@1: 73.03 (PE-VIDEO), 63.30 (VATEX), 40.75 (DREAM-1K) - vs. SigLIP2-G-OPT: 47.55, 61.50, 27.52 - vs. PECoreG (teacher): 63.91, 72.10, 18.90

Video captioning (Table 3, V-SONAR + OmniSONAR Decoder): - PE-VIDEO BLEU: 39.0 vs. 30.0 (Qwen2.5-VL-3B, best baseline) - DREAM-1K BLEU: 23.9 vs. 19.6 (PLM-3B, best baseline) - VATEX BLEU: 26.7 (below InternVL2-1B at 47.8 — expected, short captions) - VATEX-zh BLEU: 30.6 (vs. InternVL2.5-1B 33.2, but higher on ROUGE/BERTScore)

LCM zero-shot video captioning (Table 5): - Trails PLM-8B by 1.15/4.44/4.76 BLEU on PE-VIDEO/DREAM-1K/VATEX without any video training data.

V-LCM on M3IT: - Competitive with InternVL, Qwen-VL, PLM on image/video captioning and QA. - Outperforms all tested VLMs in 61/62 M3IT languages.

Limitations

  • V-SONAR degrades on short-caption benchmarks (VATEX), attributed to training predominantly on detailed caption data; not explicitly remedied.
  • Full multilingual V-LCM evaluation is M3IT-only; downstream performance on dedicated multilingual QA/retrieval benchmarks is not reported.
  • V-LCM requires V-SONAR1 (aligned to SONAR1) due to LCM pre-training, precluding use of the stronger OmniSONAR space for V-LCM experiments.
  • The SONAR1 embedding space is reported to be "collapsed" (low norm ~0.264, trace ~0.049), limiting alignment quality relative to OmniSONAR.
  • Contrastive loss addition yielded no significant gain over MSE alone, leaving potential retrieval improvements unexplored.
  • The model has not been evaluated on image-heavy benchmarks (VQA, MMMU, SeedBench), making comparison with mainstream VLM leaderboards incomplete.

Relevance to Vision-Language Models

This paper challenges the dominant paradigm of discrete-token or patch-token VLMs by showing that a modality-agnostic continuous latent space can serve as a drop-in unification layer for vision and language — including zero-shot transfer from text-only pretrained models to vision tasks. For VLM researchers, V-LCM's superior multilingual generalization (61/62 languages) without language-specific visual tuning suggests that language-agnostic embedding alignment is a more scalable path to multilingual vision-language understanding than token-level fine-tuning. The post-hoc alignment methodology also offers a lightweight recipe for extending any frozen multilingual text encoder with visual grounding, relevant to ongoing work on universal representation spaces and modality fusion.