Skip to content

VITA-1.5: Towards GPT-4o Level Real-Time Vision and Speech Interaction

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

VITA-1.5 is an open-source omni multimodal LLM that integrates vision, language, and speech through a three-stage training curriculum designed to mitigate cross-modal interference. It achieves end-to-end speech-to-speech interaction without separate ASR/TTS modules, reaching competitive performance with leading open-source VLMs while adding strong speech capabilities. The work targets the GPT-4o-level goal of simultaneous high-quality vision and real-time speech interaction.

Problem

Existing MLLMs focus on vision-language integration but poorly handle speech, while speech-capable models (cascaded ASR+LLM+TTS pipelines) suffer high latency and loss of paralinguistic features. Jointly training vision and speech modalities causes inter-modal conflicts: adding speech data degrades vision performance and vice versa. Open-source models with both vision and speech at competitive quality are scarce.

Method

VITA-1.5 uses a three-stage progressive training strategy on top of an InternViT-300M visual encoder + ~350M-parameter speech encoder + Qwen2-7B LLM backbone:

  • Stage 1 (Vision-Language): Three sub-stages — (1.1) visual adapter alignment on 20% caption data (adapter only trainable); (1.2) vision understanding on full caption data (encoder+adapter+LLM trainable); (1.3) visual SFT on QA data + 20% captions.
  • Stage 2 (Audio Input): (2.1) Speech encoder trained via CTC loss on 11,000h of ASR pairs, then audio adapter integrated with LLM for transcription; (2.2) speech SFT where ~50% of text questions are replaced with TTS-generated speech versions; a modality classification head is added to the LLM output.
  • Stage 3 (Audio Output): (3.1) TiCodec trained on 3,000h text-speech data with a single codebook (1024 entries, 40Hz token rate, 24kHz waveform output); (3.2) NAR+AR speech decoder training (4 LLaMA layers each, 120M params total) on text embeddings → speech tokens, with LLM frozen to preserve multimodal performance.

Dynamic patching is applied to high-resolution images (448×448 input, 256 tokens/image); video frames are uniformly sampled at rate-adaptive intervals (4–16 frames depending on duration).

Key Contributions

  • A three-stage training pipeline that progressively introduces modalities to minimize cross-modal interference while preserving vision-language capacity.
  • End-to-end speech output via a NAR+AR decoder architecture coupled with a single-codebook TiCodec, eliminating external TTS dependence.
  • Demonstration that audio stages (2 and 3) do not degrade vision-language performance achieved in stage 1.
  • Competitive open-source omni model supporting image, video, text, and audio in a unified framework.
  • Strong bilingual (Chinese + English) ASR performance surpassing specialized speech models.

Results

Image understanding (Table 2, VITA-1.5 Stage 3 vs. open-source, Qwen2-7B backbone): - Average score 66.8 across 9 benchmarks, comparable to InternVL2 (67.3) and LLaVA-OV (not directly listed but in same tier). - MME: 2352, surpassing InternVL-Chat-1.5 (2189.6) and Cambrian (2049.9). - MMMU: 52.1; MathVista: 66.2; OCRBench: 732.

Video understanding (Table 3): - Video-MME (w/o sub): 56.1; (w/ sub): 58.7; TempCompass: 66.7 — comparable to LLaVA-OneVision and InternVL-2, below GPT-4o (71.9/77.2).

ASR (Table 4, vs. Wav2vec2-base, Mini-Omni2, Freeze-Omni, VITA-1.0): - English WER: dev-clean 3.3 / dev-other 7.2 / test-clean 3.4 / test-other 7.5 — best among compared models on all English sets. - Outperforms Freeze-Omni (dev-clean 4.2, dev-other 9.8) and Mini-Omni2 (dev-clean 4.8).

Limitations

  • Video understanding still lags significantly behind proprietary models (GPT-4o, Gemini-1.5-Pro) in Video-MME and MVBench.
  • No personalization: model does not adapt to individual user preferences during interaction.
  • No long-term memory: interaction context is limited to a single session.
  • Speech decoder trained on TTS-generated data (3,000h), which may introduce prosody artifacts from the synthetic source.
  • Mandarin ASR evaluation does not compare against VITA-1.0 on the same test sets, making improvement less directly verifiable.

Relevance to Vision-Language Models

VITA-1.5 directly addresses the VLM community's push toward omni-modal interaction by showing that a modality-progressive training curriculum can add speech I/O to a strong VLM without sacrificing image/video performance — a key challenge for deploying VLMs in conversational agents. The staged freezing strategy (freeze LLM during codec/speech-decoder training) is a practical recipe for modality expansion that other VLM practitioners can adopt. It also benchmarks the current open-source gap to GPT-4o across vision, video, and speech axes simultaneously, providing a comprehensive reference point. For VLM researchers interested in real-time interaction, the NAR+AR speech decoding design eliminates pipeline latency without requiring architecture overhaul of the visual backbone.