Mitigating Speaker Leakage in Cascaded Multi-talker ASR with Diarization-based Transcript Correction

• URL: http://arxiv.org/abs/2608.22196v1
• Date: Sun, 23 Aug 2026 03:21:45 GMT
• Title: Mitigating Speaker Leakage in Cascaded Multi-talker ASR with Diarization-based Transcript Correction
• Abstract summary: cascaded multi-talker ASR (MT-ASR) performance is often capped by speaker leakage during separation.<n>We propose a complementary pruning-based paradigm that robustly identifies and removes leakage artifacts.<n>Results on LibriMix, LibriSpeechMix, and the AMI Meeting corpus show our algorithm consistently reduces cpW ER across diverse overlap conditions.
• Score: 0.8749675983608171
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While cascaded multi-talker ASR (MT-ASR) leverages state-of-the-art foundation models, its performance is often capped by speaker leakage during separation. Prior correction strategies primarily focus on lexical re-labeling for speaker attribution. We propose a complementary pruning-based paradigm that robustly identifies and removes leakage artifacts. Our method utilizes a pre-trained speaker diarization model as a multimodal verifier to prune transcribed segments satisfying a tripartite consensus of temporal containment, lexical cross-validation, and temporal alignment. Results on LibriMix, LibriSpeechMix, and the AMI Meeting corpus show our algorithm consistently reduces cpW ER across diverse overlap conditions. Specifically, on subsets with high speaker leakage, our method achieves relative cpW ER reductions of up to 29%, highlighting its effectiveness in enhancing the reliability of cascaded MT-ASR transcripts in complex acoustic environments.

Related papers

• Contrastive Training with LLM-generated Near-Misses for Robust Code-Switching Speech Recognition [4.573909205189436]
  We propose a Point-of-Interest (POI)-aware contrastive training framework that improves recognition at CS-critical regions.<n> Experiments on CS-FLEURS (cmn-eng) and ViMedCSS (vie-eng) show consistent reductions of over 2% in both general and CS-aware error rates.
  arXiv  Detail & Related papers  (2026-06-05T07:24:39Z)
• AgentSteerTTS: A Multi-Agent Closed-Loop Framework for Composite-Instruction Text-to-Speech [31.143755362391115]
  We introduce AgentSteerTTS, a multi-agent closed-loop framework designed for intent-faithful expressive control of composite instructions.<n>An adversarial disentanglement agent mitigates speaker-emotion leakage by learning separable identity and emotion-prosody subspaces with leakage-suppressing regularization.<n>Experiments on a composite-instruction benchmark and public test sets show that AgentSteerTTS yields consistent and significant improvements to the baselines.
  arXiv  Detail & Related papers  (2026-05-14T13:31:23Z)
• Who Spoke What When? Evaluating Spoken Language Models for Conversational ASR with Semantic and Overlap-Aware Metrics [99.89493037369071]
  Conversational automatic speech recognition remains challenging due to overlapping speech, far-field noise, and varying speaker counts.<n>Recent LLM-based systems perform well on single-speaker benchmarks, but their robustness in multi-speaker settings is unclear.<n>We systematically compare LLM-based and modular approaches along four axes: overlap, semantic fidelity, speaker count, and single- versus multi-channel input.
  arXiv  Detail & Related papers  (2026-03-24T02:01:21Z)
• StutterFuse: Mitigating Modality Collapse in Stuttering Detection with Jaccard-Weighted Metric Learning and Gated Fusion [0.40105987447353786]
  Stuttering detection breaks down when disfluencies overlap.<n>Existing parametric models struggle to distinguish complex, simultaneous disfluencies.<n>We introduce StutterFuse, the first Retrieval-Augmented generalization (RAC) for multi-label detection.
  arXiv  Detail & Related papers  (2025-12-15T18:28:39Z)
• Listen Like a Teacher: Mitigating Whisper Hallucinations using Adaptive Layer Attention and Knowledge Distillation [9.486565210140279]
  The Whisper model is widely adopted for its strong performance across multilingual and zero-shot settings.<n>Previous works to reduce hallucinations in Whisper-style ASR systems have primarily focused on audio preprocessing or post-processing of transcriptions to filter out erroneous content.<n>We present a two-stage architecture that first enhances encoder robustness through Adaptive Layer Attention (ALA) and further suppresses hallucinations using a multi-objective knowledge distillation (KD) framework.
  arXiv  Detail & Related papers  (2025-11-18T07:52:47Z)
• Towards Low-Latency Tracking of Multiple Speakers With Short-Context Speaker Embeddings [52.985061676464554]
  We propose a Knowledge Distillation based training approach for short context speaker embedding extraction.<n>We leverage the spatial information of the speaker of interest using beamforming to reduce overlap.<n>Results demonstrate that our models are effective at short-context embedding extraction and more robust to overlap.
  arXiv  Detail & Related papers  (2025-08-18T11:32:13Z)
• High-Fidelity Speech Synthesis with Minimal Supervision: All Using Diffusion Models [56.00939852727501]
  Minimally-supervised speech synthesis decouples TTS by combining two types of discrete speech representations. Non-autoregressive framework enhances controllability, and duration diffusion model enables diversified prosodic expression.
  arXiv  Detail & Related papers  (2023-09-27T09:27:03Z)
• BA-SOT: Boundary-Aware Serialized Output Training for Multi-Talker ASR [54.23941663326509]
  Frequent speaker changes can make speaker change prediction difficult. We propose boundary-aware serialized output training (BA-SOT) Compared to original SOT, BA-SOT reduces CER/UD-CER by 5.1%/14.0%.
  arXiv  Detail & Related papers  (2023-05-23T06:08:13Z)
• Continuous Speech Separation with Conformer [60.938212082732775]
  We use transformer and conformer in lieu of recurrent neural networks in the separation system. We believe capturing global information with the self-attention based method is crucial for the speech separation.
  arXiv  Detail & Related papers  (2020-08-13T09:36:05Z)
• Audio-visual Multi-channel Recognition of Overlapped Speech [79.21950701506732]
  This paper presents an audio-visual multi-channel overlapped speech recognition system featuring tightly integrated separation front-end and recognition back-end. Experiments suggest that the proposed multi-channel AVSR system outperforms the baseline audio-only ASR system by up to 6.81% (26.83% relative) and 22.22% (56.87% relative) absolute word error rate (WER) reduction on overlapped speech constructed using either simulation or replaying of the lipreading sentence 2 dataset respectively.
  arXiv  Detail & Related papers  (2020-05-18T10:31:19Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.