Skip to content

OpenOmni: Advancing Open-Source Omnimodal Large Language Models with Progressive Multimodal Alignment and Real-time Emotional Speech Synthesis

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

OpenOmni is a fully open-source omnimodal LLM that achieves cross-modal alignment across image, text, and speech via a progressive three-stage training strategy that uses text as a pivot, eliminating the need for scarce tri-modal datasets. It further adds a lightweight non-autoregressive speech decoder trained with CTC-DPO for real-time, emotionally expressive speech generation. On OmniBench, a 7B model trained on 1.6M samples outperforms VITA (7×8B, 5M samples) by 4 absolute points.

Problem

Existing open-source omnimodal LLMs face three bottlenecks: (1) high-quality image-speech-text triplet datasets are scarce and expensive, forcing reliance on suboptimal tri-modal corpora; (2) autoregressive speech generation introduces latency incompatible with real-time interaction; (3) current models produce emotionally flat speech because they lack a mechanism to align prosody with conversational context.

Method

Progressive three-stage training with text as pivot: 1. Speech-text alignment — a Whisper-large-v3 speech encoder is pretrained on 8,000 hours of bilingual speech-text pairs using a language modeling objective, grounding LLM hidden states to both speech and text representations. 2. Image-text alignment — a CLIP-ViT-L image encoder is added; the model first undergoes image-text pretraining (LLaVA-Pretrain-595K, LLM frozen) then instruction tuning (MMEvol). Because speech and text share similar hidden representations after stage 1 (f_LLM(x^{t,q}) ≈ f_LLM(h_s(x^{s,q}))), the model achieves quasi-zero-shot omnimodal understanding without any tri-modal data. 3. Text-guided speech generation — a lightweight streaming speech decoder (Qwen2.5-0.5B + MoE layer) is trained on O2S-300K (300K instructions with synthesized bilingual speech) using CTC loss. A Text-Guided Module (TGM) projects LLM hidden states to stabilize decoder training. NAR decoding is supported for sub-second latency. Emotional alignment uses CTC-DPO: preference pairs are constructed from EO2S-9K (9K bilingual multi-turn dialogues annotated with 9 Plutchik emotions), where emotion-congruent speech units are preferred over neutral/incongruent ones; only the policy model updates while the pretrained speech decoder is the frozen reference.

Key Contributions

  • Progressive omnimodal alignment framework using text as a cross-modal pivot; enables (near) zero-shot vision-to-speech transfer without tri-modal corpora.
  • O2S-300K: 8,000 hours of bilingual synthesized speech dialogue data for speech decoder training.
  • EO2S-9K: bilingual multi-turn emotional speech preference dataset covering 9 emotion categories for DPO.
  • CTC-DPO algorithm adapting DPO to discrete speech unit sequences for self-aware emotional speech generation.
  • End-to-end streaming speech decoder with both AR and NAR modes, achieving <1 s latency in NAR mode.

Results

  • OmniBench (omnimodal): OpenOmni 7B scores 37.40 overall vs. VITA 7×8B at 33.45 and VITA-1.5 7B at 33.48 — a 4-point absolute improvement using 3× fewer training samples (1.6M vs. 5M).
  • Vision-language benchmarks: Outperforms VITA (7×8B) among fully open-source models; +7.0% on MMBench-CN, +11.3% on HallusionBench; competitive with weight-open models such as Qwen2-VL-7B on several benchmarks.
  • Speech-language (WER): Best among omnimodal LLMs on AIShell-2 (ZH S2T: 6.8 CER, T2S: 7.3) and LibriSpeech test-clean (EN S2T: 3.1 WER, T2S: 2.6).
  • Inference latency: NAR mode generates up to 30 s of speech in <1 s; >5× faster than autoregressive baselines.
  • Emotional speech (EO2S-9K): CTC-DPO improves overall emotion classification accuracy by 7.7% absolute (ZH: 57.9→70.4%; EN: 62.6→65.4%); largest gains on "Sad" (ZH: +30.5%) and "Fearful" (ZH: +23.6%).

Limitations

  • Emotional accuracy remains low for some categories (e.g., "Surprised" EN: 13.9% even after DPO), suggesting DPO with small preference datasets is insufficient for rare emotions.
  • EO2S-9K is small (9K samples); emotion categories like anger and sadness required GPT-4o-mini augmentation due to natural underrepresentation in instructional data.
  • NAR speech quality may trail AR mode; paper does not report MOS or comparative perceptual quality scores.
  • Evaluated only at 7B scale; scaling behavior of the progressive alignment strategy is not analyzed.
  • No ablation of the pivot-text strategy against true tri-modal training at equivalent data budgets, so zero-shot transfer claims rely on indirect comparison with prior work.

Relevance to Vision-Language Models

OpenOmni demonstrates that a strong VLM backbone can serve as the substrate for omnimodal extension without sacrificing vision-language performance — in fact, the added speech modality appears to improve VL benchmarks (HallusionBench +11.3% over VITA). The progressive alignment strategy, which treats speech as a late-binding modality aligned through shared text representations, is directly relevant to researchers studying efficient VLM adaptation and cross-modal generalization. The work also highlights that DPO-style preference optimization, already established for VLM hallucination reduction, transfers naturally to speech output alignment. For VLM researchers, it offers a data-efficient recipe for extending image-text models to full omnimodal capability.