Skip to content

Watch and Listen: Understanding Audio-Visual-Speech Moments with Multimodal LLM

🕒 Published (v1): 2025-01-01 · Source: NeurIPS · Venue: NeurIPS 2025 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

TriSense is a triple-modality MLLM (visual, audio, speech) for video temporal understanding that uses a Query-Based Connector to dynamically reweight modality contributions based on the input query. It is paired with TriSense-2M, a 2M-sample dataset of long-form videos (~905s avg) with explicit modality-dropout annotations. Together they advance segment captioning and moment retrieval across arbitrary modality combinations.

Problem

Existing video MLLMs either process only visual inputs or naively fuse all modalities without adapting to which signals are actually informative for a given query. They also fail gracefully when one or more modalities are absent (modality dropout), a common real-world condition. Existing datasets (VAST-27M, LongVALE) assume all modalities are always present, use short clips, and concatenate unimodal captions without cross-modal reasoning.

Method

TriSense encodes visual frames (64 sampled), paired 2-second audio segments, and speech tokens through three separate pretrained expert encoders (visual, BEATs audio, speech), then compresses each to 16 tokens via Slot-Based Compression. A Query-Based Connector passes each modality's compressed tokens through a cross-attention layer against the query encoding, yielding query-relevant features per modality. Global average pooling over each modality's sequence produces compact vectors that are concatenated and fed into a single MLP to produce softmax-normalized scalar weights (w_v, w_a, w_s). The weighted, query-relevant modality features are concatenated, slot-compressed again, and refined by a two-layer MLP before entering the LLM backbone. A character-level Time Encoder represents timestamps as 6-token sequences. Causal event prediction conditions next-event generation on prior events, query, and multimodal context; a special <sync> token switches between a time head and LM head for timestamp vs. text output.

TriSense-2M is built via an automated pipeline: unimodal captions (visual, audio, speech) are synthesized into cross-modal AVS/AV/VS captions by a fine-tuned Qwen2.5-72B Generator, then scored 0–5 by a Qwen2.5-72B Judger; samples scoring ≥3 are retained. Starting from 5M samples, 2M high-quality examples across ~38K long videos are retained. Modality dropout is explicitly simulated during training.

Key Contributions

  • TriSense-2M: 2M curated annotations from ~38K long-form videos (avg 905s, ~4Ă— longer than LongVALE), with modality-dropout support across all V/A/S combinations; automated Generator–Judger LLM pipeline seeded by GPT-o1 references.
  • Query-Based Connector: softmax-weighted adaptive fusion that emphasizes the most query-relevant modality at inference time, enabling robust performance even when modalities are missing.
  • Eight-task unified framework: segment captioning and moment retrieval across all combinations {V, AV, VS, AVS} evaluated in a single model.
  • Causal event prediction with <sync> token for unified timestamp/text generation.

Results

Segment Captioning (TriSense-2M, CIDEr highlights): - AVS-SC: TriSense 8.3 vs. LongVALE 4.9, Qwen2.5-Omni 1.7 - AV-SC: 15.4 vs. LongVALE 5.9, TRACE 2.6 - V-SC: 36.3 vs. TRACE 9.5 (best prior), LongVALE 0.9

Moment Retrieval (TriSense-2M, IoU=0.5): - AVS-MR: 1.12 vs. Qwen2.5-Omni 0.61, TRACE 0.39 - VS-MR: 0.80 vs. Qwen2.5-Omni 0.61 - AV-MR: 0.57 vs. Qwen2.5-Omni 0.28

Zero-shot on public benchmarks (64 frames): - Charades-STA IoU=0.7: TriSense 27.6 vs. TRACE-uni 21.0, NumPro-FT 20.6 (best prior) - ActivityNet-Captions IoU=0.7: TriSense 27.2 vs. TRACE-uni 24.7, TRACE 24.0 - LongVALE benchmark (zero-shot): AVS-MR R@0.5 9.3 vs. LongVALE* (trained on dataset) 8.6

Ablations: - Query-adaptive weighting outperforms fixed-equal-weight and simple addition across all tasks - Increasing frames 64→128 yields consistent gains; V-only MR slightly lags visual-specialist models at 64 frames

Limitations

  • Uses only 64 frames at inference; performance on long videos is frame-budget-constrained (TRACE and VTimeLLM use 128–100 frames), making AVS-MR numbers artificially suppressed.
  • Visual-only moment retrieval slightly underperforms visual-specialist baselines, attributed to optimization for multimodal settings.
  • Zero-shot segment captioning on LongVALE lags the in-domain trained LongVALE model, attributed to caption-style domain shift.
  • Dataset construction relies on automated LLM-based quality scoring; human evaluation of annotation quality is not reported.
  • No evaluation of audio-only or speech-only modality configurations.

Relevance to Vision-Language Models

TriSense directly advances the frontier of video VLMs by demonstrating that query-conditioned modality weighting is essential for robust tri-modal temporal understanding—a step beyond standard vision-language alignment. The Query-Based Connector's cross-attention + softmax-weighting design is a lightweight, transferable mechanism that could inform how future VLMs handle heterogeneous or missing input streams. The TriSense-2M dataset, with its 905-second average video length and explicit modality-dropout protocol, establishes a new resource for training and evaluating long-video MLLMs beyond the visual-only paradigm. This work signals that scaling VLMs to real-world robustness requires not just more modalities but architectures that know when to attend to each.