Evaluating Large Language Models Abilities for Addressee, Turn-change, and Next Speaker Prediction in Meetings¶

🕒 Published (v1): 2026-06-16 05:46 UTC · Source: Arxiv · Venue: INTERSPEECH 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

TL;DR¶

This paper benchmarks text-based LLMs and multimodal LLMs (MM-LLMs) against supervised baselines and human subjects on three turn-taking tasks in four-party meeting conversations: addressee detection, turn-change prediction, and next speaker prediction. Text-based LLMs (Qwen3-14B) outperform humans on next speaker prediction without any audio-visual input, while the best MM-LLM (Gemini 2.5 Pro) surpasses text-only LLMs on addressee and turn-change tasks but stays below human performance, revealing that current MM-LLMs cannot reliably exploit raw audio-visual signals.

Problem¶

Prior work on turn-taking in multi-party conversations (MPCs) either targets a single task, relies on limited input modalities, or lacks systematic comparison across supervised models, text-based LLMs, MM-LLMs, and human subjects under a unified, online (no future context) protocol. Additionally, quantitative human performance under comparable multimodal conditions was unknown, making it impossible to calibrate model progress.

Method¶

The authors build an evaluation framework on the AMI Meeting Corpus (10 sessions, 4,051 utterances, 4-speaker meetings with synchronized headset audio, video, and manual transcripts). Two primary tasks are defined: utterance-level addressee detection (\(y_i \in \{A, B, C, D, \text{Group}, \text{None}\}\)) and next-speaker prediction (\(\hat{S}_{i+1} \subseteq P\)), from which turn-change labels are derived. Models receive only past and current utterance information (online constraint). Four supervised models (Naive Bayes, Random Forest, MLP, SVM) use text-derived features; Qwen3 variants (8B/14B/32B) are used as text-only LLMs with thinking mode; Qwen2.5-Omni-7B, Qwen3-Omni-30B, and Gemini 2.5 Pro serve as MM-LLMs receiving speaker ID, ground-truth transcription, utterance-level audio clip, and video clip with overlaid participant labels. A 12-subject human evaluation uses a web interface that pauses video at utterance boundaries for online annotation; each utterance receives three independent labels.

Key Contributions¶

First unified, online, multimodal benchmark comparing supervised models, text-based LLMs, MM-LLMs, and humans on all three MPC turn-taking tasks (addressee, turn-change, next speaker) under a single protocol.
Quantitative human performance baselines under multimodal online conditions (Krippendorff's \(\alpha = 0.67\) for addressee, \(0.48\) for next speaker).
Demonstration that text-based LLMs outperform humans on next speaker prediction despite lacking audio-visual input, with conversational context identified as the critical factor.
Evidence that Gemini 2.5 Pro benefits from raw audio-visual input on addressee and turn-change tasks but still falls short of human-level performance.
Ablation analyses on context window length, ASR vs. ground-truth transcription, FOA label injection, and LLM thinking mode.

Results¶

Next speaker prediction (full set, F1): Qwen3-14B 51.1, Qwen3-32B 49.7; Gemini 2.5 Pro 47.7; SVM 40.1; naive baseline 25.0.
Next speaker prediction (human-eval subset): Human F1 = 60.1; Qwen3-14B F1 = 69.4 (significantly better, paired bootstrap 95% CI); Gemini 2.5 Pro F1 = 57.7 (not significantly different from human).
Addressee detection (subset, Acc): Human 66.6%; Gemini 2.5 Pro 61.4% (not significantly below human); Qwen3-14B 51.5% (significantly below human).
Turn-change prediction (subset, Acc): Human 75.0%; Gemini 2.5 Pro 69.8% (significantly below human); Qwen3-14B 67.4% (significantly below human).
Qwen3-Omni-30B collapsed on most tasks (addressee Acc 32.1%, next speaker F1 13.7%), performing near or below naive baseline, indicating training instability on this domain.
Providing explicit FOA gaze labels (instead of raw video) improved addressee detection for MM-LLMs, suggesting models fail to extract gaze cues from raw video.
Removing conversational context degraded next speaker prediction substantially more than addressee detection.

Limitations¶

Human evaluation used 12 non-native English speakers, likely underestimating true human performance.
Human subjects could retain at most ~5 minutes of prior context, while models received full preceding transcripts within the session chunk.
Audio leakage between headset microphones (observed in ~8.5% of utterances) may make next-speaker partially observable from audio, partially inflating MM-LLM audio-based scores.
Evaluation is restricted to a single corpus (AMI) with scripted scenario meetings of exactly four participants; generalizability to spontaneous or differently-sized groups is untested.
ASR transcription experiments show performance drops, but the primary results use ground-truth transcripts, which are unavailable in real deployments.
The Qwen3-Omni-30B instability was not fully diagnosed; the thinking-mode-only variant was used.

Relevance to Vision-Language Models¶

This paper directly probes the ability of current MM-LLMs to leverage raw video streams for fine-grained conversational understanding, a core challenge for video-language models. The finding that Gemini 2.5 Pro benefits from audio-visual input for addressee and turn-change prediction—yet still cannot close the gap with humans—provides concrete evidence that state-of-the-art VLMs struggle to extract socially informative visual cues (gaze direction, head orientation, gesture) from continuous video without explicit symbolic grounding. The ablation showing that substituting raw video with annotated FOA labels recovers performance points to a bottleneck in visual grounding of social signals, a gap directly relevant to multimodal social-scene understanding research. The benchmark itself (AMI corpus, online prediction protocol, human baselines) offers a reusable testbed for measuring VLM progress on dynamic, multi-speaker video understanding.