Skip to content

Empowering Long-form Omni-modal Understanding with Robust Audio Perception

🕒 Published (v1): 2026-07-11 13:04 UTC · Source: Arxiv · link

Why this paper was selected

Weidi Xie (SJTU/Oxford); long-form omni-modal understanding with explicit audio alignment; scarce labeled data angle

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

Current multimodal LLMs suffer from systematic audio neglect because training corpora lack dense, modality-independent auditory supervision. This paper introduces AVDC, a 10K-video dataset with automatically generated tripartite captions (audio-only, visual-only, joint audio-visual) plus AVDC-QA-CoT with chain-of-thought rationales, and a two-stage training paradigm on top of Qwen-2.5-Omni-7B-Thinker. The resulting model shows significant gains on omni-modal captioning and audio-centric benchmarks compared to strong baselines.

Problem

Vision-centric MLLMs fail in audio-primary scenarios (off-screen sirens, prosody inference, blender vs. dishwasher) because: (1) audio supervision in existing datasets is dominated by speech transcripts, neglecting non-speech environmental sounds; and (2) sound events are typically synchronized with visible objects ("visual redundancy"), allowing models to predict acoustic events from visual features alone, making the audio modality effectively auxiliary.

Method

Dataset construction (AVDC): An automated pipeline processes 10K videos from ShareGPT4Video and Vript. Videos are filtered to contain ≥3 distinct environmental sound events (density >10% of duration). Each video is segmented at audiovisual transitions into segments \(S = \{s_1, \ldots, s_n\}\) of 2–30 s. For each segment, four specialist models extract raw audio information: Qwen2-Audio and Audio-Flamingo 2 (general audio; cross-verified with Cohen's \(\kappa\)), SALMONN (speech), MU-LLaMA (music). A hierarchical LLM fusion then generates segment-level audio captions \(\mathcal{C}^a_k\) and joint captions \(\mathcal{C}_k\) conditioned on the preceding segment, and a global LLM refinement aggregates into video-level \(\mathcal{C}^a_g, \mathcal{C}_g\). Iterative verification-correction by Gemini-2.5-Flash enforces a quality score ≥4/5 before retention.

QA dataset (AVDC-QA-CoT): DeepSeek generates QA pairs from sampled captions with structured four-step CoT rationales: (i) question decomposition, (ii) temporal grounding, (iii) modality-specific cue extraction, (iv) multimodal integration. Samples answerable from text alone (Qwen3-4B correctly answers without video, or METEOR >0.2 for open-ended) are discarded.

Training: Stage I freezes the LLM backbone and fine-tunes visual and audio encoders on AVDC for global and grounded (timestamped) omni-modal captioning. Stage II performs instruction tuning on AVDC-QA-CoT augmented with audio-visual misalignment examples (injected out-of-context sounds) and balanced video/audio QA benchmarks, using dual-mode supervision (thinking with <think>...</think> and non-thinking). Backbone: Qwen-2.5-Omni-7B-Thinker with TMRoPE for joint temporal-spatial-audio positional encoding.

Key Contributions

  • AVDC dataset: 10K long-form videos (~30 s avg) with tripartite decoupled captions averaging 258 tokens; 5.5 sound events/video with 37% invisible (off-screen), substantially exceeding existing datasets (prior best: 2.3 total events, 0.5 invisible in UGC-VideoCap).
  • AVDC-QA-CoT: 10K multimodal QA pairs with structured four-step CoT rationales covering temporal, spatial, causal, and cross-modal reasoning; filtered to require genuine multimodal evidence.
  • Automated annotation pipeline combining multi-model audio extraction, hierarchical LLM fusion, and iterative Gemini-2.5-Flash verification; sound-event F1 improves from 0.467 (single model) → 0.671 (merged) → 0.786 (+ visual cues) → 0.885 (+ second verification).
  • Two-stage training paradigm with controlled audio-visual misalignment augmentation to prevent vision-text shortcuts.
  • AVDC-test benchmark for evaluating visible vs. invisible sound event recovery in generated captions.

Results

Only pipeline-quality and qualitative results are present in the provided text; final benchmark tables are truncated. Available numbers:

  • Individual AudioLLM F1 scores: Qwen2-Audio = 0.467, Audio-Flamingo 2 = 0.449 (against manually annotated ground truth on 300 samples; inter-model \(\kappa = 0.134\)).
  • After DeepSeek merging: F1 = 0.671; after Gemini-2.5-Flash verification: 0.752.
  • Adding visual cues raises sound-event F1 to 0.786; second verification pass: 0.885.
  • Vision-event F1: 0.842 (pre-verification) → 0.921 (post Gemini verification).
  • Authors claim "consistent and significant performance gains" on AVDC-test, Video-SALMONN-2, UGC-VideoCap, WorldSense, Omnibench, and MMAU vs. baselines including Qwen2.5-Omni and HumanOmniV2 (qualitative comparison in Fig. 1 shows fewer hallucinations and missing events).

Limitations

  • Dataset scale (10K videos) is orders of magnitude smaller than VALOR-1M or VAST-27M, which may limit generalization to rare acoustic domains.
  • Automated pipeline relies on models with substantial individual biases (\(\kappa = 0.134\) between audio models); post-hoc verification partially compensates but cannot fully eliminate systematic errors.
  • The "visual redundancy" problem is mitigated by filtering, not structurally solved; invisible-event coverage (37%) still leaves 63% as visually corroborated events.
  • Audio-visual misalignment augmentation uses out-of-context sound injection—a synthetic perturbation that may not reflect naturalistic audio degradation.
  • CoT rationales are LLM-generated from captions, not from raw video, introducing a caption-mediated supervision gap.
  • 7B-parameter backbone; scaling behavior unknown.

Relevance to Vision-Language Models

This paper directly targets a structural blind spot in VLM training: audio is treated as redundant to vision in most VLM corpora, preventing genuine multimodal reasoning. The tripartite caption decoupling strategy (V / A / AV) is a dataset-design principle transferable to any VLM training pipeline seeking to disentangle modality contributions. The two-stage curriculum—first aligning audio-visual representations before instruction tuning—is a practical recipe for extending vision-language backbones to omni-modal settings without catastrophic forgetting of visual competence. AVDC-QA-CoT's structured CoT (decompose → ground → perceive → integrate) aligns with the broader push for chain-of-thought supervision in VLMs and could inform multimodal reasoning benchmarks beyond audio.