Skip to content

Reasoning LLM Improves Speaker Recognition in Long-form TV Dramas

🕒 Published (v1): 2026-07-02 17:58 UTC · Source: Arxiv · Venue: ICML 2026 · link

Ask a follow-up

Open an assistant pre-loaded with this paper's context.

💬 Ask ChatGPT✦ Ask Claude

TL;DR

This paper introduces DramaSR-532K, a 532K-utterance benchmark for speaker recognition in long-form TV dramas, and DramaSR-LRM, a reasoning LLM trained with SFT+RL that autonomously invokes multimodal tools (voiceprint similarity, video captioning, character relations) to attribute dialogue lines to characters. The system improves over a label-propagation baseline by 2.30% overall and up to 9.20% on very short utterances where acoustic biometrics fail.

Problem

Existing speaker diarization/verification benchmarks (AVA-AVD: 2–24 speakers, short clips) do not capture the scale and complexity of long-form TV dramas: 30–100+ named characters per series, hours of content per episode, off-screen speech, overlapping dialogue, and short utterances where voiceprints are unreliable. No large-scale, accurately annotated benchmark or reasoning-centric solution existed for this setting.

Method

DramaSR-532K construction: Subtitle text is extracted via OCR (PaddleOCR + Qwen2.5VL refinement); a character library is built from credits and in-situ facial samples collected at 1 FPS. Label propagation initializes annotations: acoustic embeddings (ERes2Net, 192-D, \(\ell_2\)-normalized) are clustered with a greedy step-wise procedure and refined by affinity propagation using a spatiotemporal neighborhood assumption (\(\tau = 30\text{s}\)). Human annotators then audit and correct pseudo-labels; inter-annotator agreement is 99.6% with estimated final label noise ~0.5%.

DramaSR-LRM: A Qwen3-8B backbone is trained in two stages: 1. SFT on Gemini-3-Pro-generated chain-of-thought trajectories that interleave tool calls with reasoning over a 20–30 line dialogue context window. 2. RL (GRPO) to optimize reasoning consistency.

The model has access to three tools: voice_sim (cosine similarity matrix \(K \in \mathbb{R}^{N \times P}\) against known voiceprint seeds), video_cap (MLLM-based visual scene description), and char_relation (relational DB encoding social hierarchies and appellations). The system runs iteratively: tool outputs are updated with refined labels from the prior round until convergence.

Key Contributions

  • DramaSR-532K: 806 episodes, 525h31m, 532K annotated utterances, 900+ primary and 6,600+ secondary characters across 13 TV series (English and Chinese), with a rigorous human-in-the-loop annotation pipeline.
  • DramaSR-LRM: First LRM applied to TV drama speaker recognition, trained via SFT+RL with multimodal tool-use.
  • Demonstrated gains specifically on hard cases: short/very-short utterances and challenging dramas with many overlapping characters.
  • Public release of data and code.

Results

  • Overall accuracy improvement over label-propagation baseline: +2.30% (85.49% → 87.79%) across 11 held-out dramas.
  • Short utterances: +3.33% gain.
  • Very short utterances: +9.20% gain.
  • Challenging dramas (low baseline): Lost +5.16%, Qin Empire 2 +4.06%.
  • Label propagation baseline itself achieves ~85–90% accuracy as initialization; DramaSR-LRM improves atop this without degrading easier cases.

(Note: comparisons are against the label-propagation baseline; no other LRM/MLLM baselines are reported in the provided text.)

Limitations

  • The spatiotemporal neighborhood assumption (\(\pm 30\text{s}\) visual window) biases the candidate set toward visually anchored characters, potentially underrepresenting truly off-screen speech difficulty.
  • Training uses only a single drama (50K utterances); generalization to stylistically diverse dramas may vary.
  • The SFT teacher is Gemini-3-Pro—distillation quality depends on the teacher's accuracy; incorrect teacher outputs are re-prompted with ground truth rather than discarded.
  • Evaluation assumes pre-segmented, timestamped utterances; end-to-end recognition (segmentation + attribution) remains an open problem flagged as future work.
  • Open-world speaker description (generating textual descriptions for unknown characters) is not yet addressed.

Relevance to Agentic AI / LLM Agents

DramaSR-LRM is a direct instantiation of a tool-using reasoning agent: the LRM autonomously decides which tools to invoke, in what order, and synthesizes their outputs into a final decision—precisely the agentic loop being studied in LLM agent research. The SFT+RL training paradigm for instilling tool-use behavior and chain-of-thought consistency is directly analogous to techniques used in general-purpose LLM agents (e.g., GRPO-based policy refinement). The iterative refinement loop where updated labels feed back into tool outputs also mirrors the memory-and-action cycles discussed in long-horizon agentic frameworks. For researchers tracking agentic AI, this paper provides a concrete, evaluated case study of an RL-trained tool-use agent on a structured multimodal task with verifiable ground truth.