Skip to content

ThinkSound: Chain-of-Thought Reasoning in Multimodal LLMs for Audio Generation and Editing

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

ThinkSound introduces a three-stage video-to-audio (V2A) generation framework that uses Chain-of-Thought (CoT) reasoning from a fine-tuned multimodal LLM to guide a unified flow-matching audio foundation model. It decomposes sound design into foley generation, click-based object-centric refinement, and instruction-based editing. The framework is supported by AudioCoT, a new large-scale dataset with structured audio reasoning annotations.

Problem

Existing end-to-end V2A systems (e.g., MMAudio, FoleyCrafter) compress complex audio reasoning into a single black-box transformation, failing to capture temporal event ordering, causal acoustic relationships, and subtle visual cues (e.g., distinguishing an owl chirping vs. flapping). Prior MLLM-based approaches either discard visual detail by converting to text captions first (SonicVisionLM) or fragment the pipeline across specialized models (DeepSound-V1), preventing unified, interactive audio design.

Method

ThinkSound fine-tunes VideoLLaMA2 on the AudioCoT dataset to generate structured CoT reasoning chains capturing temporal dependencies, acoustic properties, and causal relationships. These CoT instructions condition a unified MM-DiT audio foundation model trained with conditional flow matching. The foundation model uses: - Dual-pathway text encoding: MetaCLIP for global scene context + T5-v1-xl for fine-grained CoT reasoning. - Gated adaptive fusion: upsampled video features are merged with audio latents via a gated mechanism before single-stream transformer blocks. - Classifier-free guidance dropout (p=0.2 per modality) to handle arbitrary input combinations at inference. - Three inference stages: (1) full-video foley generation, (2) ROI-based object-centric refinement using Grounded SAM2 click prompts, (3) natural-language instruction-based editing (add/remove/inpaint/extend).

AudioCoT is constructed via an automated pipeline using VideoLLaMA2 + Qwen2-Audio + GPT-4.1-nano, sourcing from VGGSound, AudioSet, AudioCaps, Freesound, and BBC Sound Effects.

Key Contributions

  • Three-stage interactive V2A framework unified by MLLM-generated CoT reasoning.
  • Unified multimodal audio foundation model (1.30B params) supporting arbitrary video/text/audio input combinations via flow matching.
  • AudioCoT: large-scale dataset with audio-specific CoT annotations covering foley generation, object-centric refinement, and editing operations.
  • Click-based interface integrating Grounded SAM2 for ROI-targeted audio synthesis.

Results

VGGSound test set (vs. best baseline MMAudio): - FD: 34.56 vs. 39.15 (V-AURA best); ThinkSound best overall - KLPaSST: 1.52 vs. 1.65 (MMAudio) - KLPaNNs: 1.32 vs. 1.40 (MMAudio) - CLAPCoT: 0.46 vs. 0.40 (MMAudio) - MOS-Q: 4.02±0.73 vs. 3.84±0.89 (MMAudio) - MOS-A: 4.18±0.79 vs. 3.97±0.82 (MMAudio) - Inference: 1.07s vs. 3.01–19.42s for baselines

MovieGen Audio Bench (OOD): - CLAPCoT: 0.51 vs. 0.47 (MovieGen), MOS-Q: 4.11, MOS-A: 3.87 (best)

Object-focused generation (Table 3): ThinkSound CLAP 0.48 vs. MMAudio 0.41, MOS-A 3.91 vs. 3.64.

Audio editing (Table 4): ThinkSound FD 34.78, CLAP 0.51 vs. DDPM (55.56, 0.39).

Ablation: Removing CoT drops CLAPCoT from 0.46→0.41 and FD from 34.56→39.84; gated video fusion outperforms linear addition and audio-only across all metrics.

Limitations

  • Authors acknowledge "challenges remained in capturing nuanced acoustic properties" (e.g., subtle timbre, room acoustics).
  • No physical acoustic modeling; planned for future work.
  • Object-centric stage requires user click interaction, limiting full automation.
  • AudioCoT construction relies on GPT-4.1-nano synthesis, potentially introducing annotation noise.
  • DeSync metric for OOD evaluation (0.76) is on par with MMAudio (0.77) but not clearly superior.

Relevance to Vision-Language Models

ThinkSound demonstrates that CoT reasoning—central to recent MLLM advances—transfers effectively to cross-modal generation beyond text, bridging visual understanding and audio synthesis in a way directly analogous to VLM-guided image generation. The fine-tuning of VideoLLaMA2 to produce domain-specific reasoning chains (audio-centric CoT) is a methodological template applicable to any VLM tasked with guiding a downstream generative model. The dual-encoder conditioning (CLIP for global context + T5 for structured reasoning) mirrors design patterns seen in vision-language grounding architectures. For VLM researchers, this work establishes that structured intermediate reasoning from MLLMs meaningfully improves fine-grained spatiotemporal alignment in generation tasks, a finding with direct implications for video understanding and multimodal instruction following.