JavisGPT: A Unified Multi-modal LLM for Sounding-Video Comprehension and Generation¶

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

JavisGPT is the first unified MLLM that jointly comprehends and generates synchronized audio-video (sounding video) content from multimodal instructions. It introduces a SyncFusion module for spatio-temporal audio-video fusion and bridges a pretrained JAV diffusion transformer (JavisDiT) via hierarchical learnable query embeddings. A three-stage training pipeline progressively adapts a Qwen2.5-VL-7B backbone to these joint tasks.

Problem¶

Existing MLLMs treat audio and video as independent modalities—either concatenating or interleaving features—without modeling fine-grained spatio-temporal synchrony between them. Pipeline-based generation systems (NExT-GPT, UnifiedIO-2) suffer from error propagation and desynchronized outputs. No prior MLLM unifies sounding-video comprehension and generation in a single model with explicit cross-modal temporal grounding.

Method¶

JavisGPT uses an encoder–LLM–decoder architecture on top of Qwen2.5-VL-7B-Instruct. Video frames are encoded with Qwen2.5-VL's vision encoder (yielding V ∈ R^{Tv×(H×W)×C}); audio waveforms are converted to mel-spectrograms and encoded with BEATs, then projected via a 2-layer MLP into the LLM token space.

SyncFusion temporally aligns audio segments to video frames, then applies a shared cross-attention module to inject audio cues into frame-wise visual patches, producing SyncAV tokens (one token per spatial patch per frame), preserving MRoPE positional encoding compatibility.

For generation, N learnable JavisQuery tokens (Qc) are inserted into the LLM sequence on encountering a special <|av_start|> token; their hidden states are projected via MLP to match the condition space of the frozen JavisDiT (T5-XXL space), with an alignment loss ‖ĉ−c‖². Additional N′ ST-prior queries (Qs) estimate spatial (where) and temporal (when) event priors, aligned with JavisDiT's ST-prior encoder. The composite loss is Lcomp = Lcalign + Lsalign + Ldiff.

Three-stage training: (1) MM-PreTrain: align audio MLP and Qc/ϕc with 600K audio-text + 1.5M AV captions; (2) AV-FineTune: train SyncFusion + LoRA (r=128) on 360K AV triplets for synchrony; (3) MM-InstTune: joint instruction tuning on JavisInst-Omni (200K+ GPT-4o-curated dialogues covering QA, captioning, interleaved generation, multi-turn conversations).

Key Contributions¶

First unified MLLM for joint audio-video (JAV) comprehension and generation in a single model.
SyncFusion module: cross-attention-based spatio-temporal fusion producing per-patch SyncAV tokens, achieving better AV-understanding performance at lower token count (2.0K vs. 3.5K) and lower latency (224ms vs. 263–555ms for alternatives).
Hierarchical JavisQuery design: separate semantic (Qc) and ST-prior (Qs) conditioning for the frozen JAV-DiT, avoiding full DiT fine-tuning while enabling synchronized generation.
JavisInst-Omni: 200K+ GPT-4o-annotated, manually verified multimodal instruction-tuning dataset spanning comprehension and generation tasks.
Three-stage progressive training pipeline demonstrating that staged adaptation is necessary (skipping MM-PreTrain forces 10× LR reduction and yields worse results).

Results¶

Audio-video comprehension (Tab. 2, 7B models): - AVQA: 93.8 (vs. VideoLLaMA2.1: 91.5, Qwen2.5-Omni: 80.9) - MU-AVQA: 82.1 (vs. VideoLLaMA2.1: 80.9, Qwen2.5-Omni: 79.9) - AVSD: 62.2 (vs. Qwen2.5-Omni: 62.8, VideoLLaMA2.1: 57.2) - MVBench: 68.4 (vs. Qwen2.5-Omni: 66.9); TUT2017 audio: 82.1 (vs. Qwen2.5-Omni: 78.3)

Audio-video generation (Tab. 3, JavisBench-mini): - FVD: 243.6 (vs. JavisDiT: 248.9; NExT-GPT: 1463.2; UnifiedIO-2: 1597.4) - KVD: 1.8 (vs. JavisDiT: 1.9; NExT-GPT: 6.1) - JavisScore: 0.157 (vs. JavisDiT: 0.153; NExT-GPT: 0.038; UnifiedIO-2: 0.053) - AV-IB (AV consistency): 0.202 (vs. JavisDiT: 0.203; NExT-GPT: 0.067)

Human evaluation (interleaved comprehension+generation, 100 multi-turn dialogues): JavisGPT outperforms NExT-GPT and UnifiedIO-2 on all five dimensions (QA, instruction following, context reasoning, proactive thinking, generation quality).

Efficiency: Trained with only 1.5M total samples vs. 1.9M+ for VideoLLaMA2/VideoLLaMA2.1.

Limitations¶

AV synchrony improvement from joint training is marginal (+0.04), suggesting the current SyncFusion–generation coupling is not fully bidirectional.
No speech input/output support; BEATs processes non-speech audio only; replacing with Whisper/WavTokenizer is left as future work.
JavisDiT generator is frozen; full end-to-end training with a unified autoregressive model (discrete+continuous tokens) is not explored.
SyncFusion reduces but does not eliminate token redundancy (2.0K vs. 0.2K for Q-Former), trading some efficiency for stability.
Evaluation of interleaved generation uses a curated 100-sample human evaluation rather than a standardized automated benchmark.

Relevance to Vision-Language Models¶

JavisGPT directly extends the VLM paradigm—specifically Qwen2.5-VL—to a third modality (audio) while preserving the vision-language backbone's spatial and temporal reasoning strengths, making it a case study in minimally invasive multimodal extension of strong VLM foundations. The SyncFusion design offers a blueprint for how to incorporate a new modality into an existing VLM without disrupting positional encoding or forcing full fine-tuning. The finding that joint comprehension–generation training mutually improves both sides has implications for any VLM being extended toward generative capabilities (e.g., image/video synthesis conditioned on language). The JavisInst-Omni dataset and three-stage curriculum are directly applicable to researchers building audio-augmented VLMs or video-understanding models that need synchronized multi-modal outputs.