Reasoning LLM Improves Speaker Recognition in Long-form TV Dramas

• URL: http://arxiv.org/abs/2607.02504v1
• Date: Thu, 02 Jul 2026 17:58:52 GMT
• Title: Reasoning LLM Improves Speaker Recognition in Long-form TV Dramas
• Abstract summary: DramaSR-LRM is designed to autonomously aggregate contextual evidence via multimodal tool-use, diverse inputs to achieve high-fidelity attribution.<n> Experimental results demonstrate that DramaSR-LRM significantly outperforms existing baselines.
• Score: 76.70045115464431
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-form TV dramas present a formidable challenge for comprehensive video understanding, where deciphering complex storyline often relies on \textbf{speaker recognition}, the task of accurately attributing each spoken utterance to its respective character. In this paper, we advance this field through two primary contributions. (1) We introduce \textbf{DramaSR-532K}, a large-scale benchmark comprising 532K annotated dialogue lines across more than 900 unique characters, necessitating the integration of auditory, linguistic, and visual cues for speaker recognition. (2) We propose \textbf{DramaSR-LRM}, a robust approach built upon a large reasoning model (LRM). DramaSR-LRM is designed to autonomously aggregate contextual evidence via multimodal tool-use, synthesizing diverse inputs to achieve high-fidelity attribution. Experimental results demonstrate that DramaSR-LRM significantly outperforms existing baselines, particularly on short utterances where acoustic biometrics are inherently unreliable. \textit{All the data and code will be made publicly available at the project page: https://www.github.com/198808xc/DramaSR-LRM.}

Related papers

• ATIR: Towards Audio-Text Interleaved Contextual Retrieval [63.68521448682396]
  We introduce the Audio-Text Interleaved contextual Retrieval (ATIR) task, where queries can alternate between audio and text modalities.<n>This benchmark substantially addresses the limitations of existing audio retrieval datasets in semantic retrieval.
  arXiv  Detail & Related papers  (2026-04-22T07:11:58Z)
• DialogGraph-LLM: Graph-Informed LLMs for End-to-End Audio Dialogue Intent Recognition [10.94195981338177]
  DialogGraph-LLM is an end-to-end framework for recognizing speaker intent in audio dialogues.<n>It combines a novel Multi-Relational Dialogue Attention Network (MR-DAN) architecture with multimodal foundation models for direct acoustic-to-intent inference.<n>The framework demonstrates strong performance and efficiency in intent recognition in real world scenario audio dialogues.
  arXiv  Detail & Related papers  (2025-11-14T06:42:04Z)
• VoxRole: A Comprehensive Benchmark for Evaluating Speech-Based Role-Playing Agents [25.2769510663867]
  We introduce VoxRole, the first comprehensive benchmark specifically designed for the evaluation of speech-based RPCAs.<n>The benchmark comprises 13335 multi-turn dialogues, totaling 65.6 hours of speech from 1228 unique characters across 261 movies.
  arXiv  Detail & Related papers  (2025-09-04T07:03:46Z)
• AHELM: A Holistic Evaluation of Audio-Language Models [78.20477815156484]
  multimodal audio-language models (ALMs) take interleaved audio and text as input and output text.<n>AHELM is a benchmark that aggregates various datasets -- including 2 new synthetic audio-text datasets called PARADE and CoRe-Bench.<n>We also standardize the prompts, inference parameters, and evaluation metrics to ensure equitable comparisons across models.
  arXiv  Detail & Related papers  (2025-08-29T07:40:39Z)
• MOMENTS: A Comprehensive Multimodal Benchmark for Theory of Mind [41.188841829937466]
  MoMentS (Multimodal Mental States) is a benchmark for building socially intelligent multimodal agents.<n>MoMentS includes over 2,300 multiple-choice questions spanning seven distinct ToM categories.<n>We evaluate several MLLMs and find that although vision generally improves performance, models still struggle to integrate it effectively.
  arXiv  Detail & Related papers  (2025-07-06T15:06:30Z)
• What Makes a Good Speech Tokenizer for LLM-Centric Speech Generation? A Systematic Study [58.55905182336196]
  Speech-language models (SLMs) offer a promising path toward unifying speech and text understanding and generation.<n>We investigate the role of speech tokenizer designs in LLM-centric SLMs, augmented by speech heads and speaker modeling.<n>We introduce multi-token prediction (MTP) into SLMs, enabling each hidden state to decode multiple speech tokens.
  arXiv  Detail & Related papers  (2025-06-14T15:26:31Z)
• CoVoMix2: Advancing Zero-Shot Dialogue Generation with Fully Non-Autoregressive Flow Matching [78.01028753403575]
  CoVoMix2 is a framework for zero-shot multi-talker dialogue generation.<n>It predicts mel-spectrograms from multi-stream transcriptions using a flow-matching-based generative model.<n>Our approach achieves state-of-the-art performance, outperforming strong baselines like MoonCast and Sesame in speech quality, speaker consistency, and inference speed.
  arXiv  Detail & Related papers  (2025-06-01T07:51:45Z)
• CMT-LLM: Contextual Multi-Talker ASR Utilizing Large Language Models [23.278483193586887]
  We propose a unified framework that combines multi-talker overlapping speech recognition and contextual biasing into a single task.<n>Our approach outperforms traditional contextual biasing methods, achieving a WER of 7.9% on LibriMix and 32.9% on AMI SDM.
  arXiv  Detail & Related papers  (2025-05-31T07:26:44Z)
• EchoInk-R1: Exploring Audio-Visual Reasoning in Multimodal LLMs via Reinforcement Learning [108.73513190593232]
  Multimodal large language models (MLLMs) have advanced perception across text, vision, and audio, yet struggle with structured cross-modal reasoning.<n>We introduce EchoInk-R1, a reinforcement learning framework that enhances such reasoning in MLLMs.
  arXiv  Detail & Related papers  (2025-05-07T17:59:49Z)
• Large Language Models are Strong Audio-Visual Speech Recognition Learners [53.142635674428874]
  Multimodal large language models (MLLMs) have recently become a focal point of research due to their formidable multimodal understanding capabilities.<n>We propose Llama-AVSR, a new MLLM with strong audio-visual speech recognition capabilities.<n>We evaluate our proposed approach on LRS3, the largest public AVSR benchmark, and we achieve new state-of-the-art results for the tasks of ASR and AVSR with a WER of 0.79% and 0.77%, respectively.
  arXiv  Detail & Related papers  (2024-09-18T21:17:27Z)
• LibriheavyMix: A 20,000-Hour Dataset for Single-Channel Reverberant Multi-Talker Speech Separation, ASR and Speaker Diarization [31.01716151301142]
  We present a large-scale far-field overlapping speech dataset to advance research in speech separation, recognition, and speaker diarization. This dataset is a critical resource for decoding Who said What and When'' in multi-talker, reverberant environments.
  arXiv  Detail & Related papers  (2024-09-01T19:23:08Z)
• Disentangled Speech Embeddings using Cross-modal Self-supervision [119.94362407747437]
  We develop a self-supervised learning objective that exploits the natural cross-modal synchrony between faces and audio in video. We construct a two-stream architecture which: (1) shares low-level features common to both representations; and (2) provides a natural mechanism for explicitly disentangling these factors.
  arXiv  Detail & Related papers  (2020-02-20T14:13:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.