TOGGL: Transcribing Overlapping Speech with Staggered Labeling

• URL: http://arxiv.org/abs/2408.06474v1
• Date: Mon, 12 Aug 2024 20:19:27 GMT
• Title: TOGGL: Transcribing Overlapping Speech with Staggered Labeling
• Authors: Chak-Fai Li, William Hartmann, Matthew Snover,
• Abstract summary: We propose a model to simultaneously transcribe the speech of multiple speakers. Our approach generalizes beyond two speakers, even when trained only on two-speaker data.
• Score: 5.088540556965433
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Transcribing the speech of multiple overlapping speakers typically requires separating the audio into multiple streams and recognizing each one independently. More recent work jointly separates and transcribes, but requires a separate decoding component for each speaker. We propose the TOGGL model to simultaneously transcribe the speech of multiple speakers. The TOGGL model uses special output tokens to attribute the speech to each speaker with only a single decoder. Our approach generalizes beyond two speakers, even when trained only on two-speaker data. We demonstrate superior performance compared to competing approaches on a conversational speech dataset. Our approach also improves performance on single-speaker audio.

Related papers

• Speaker Mask Transformer for Multi-talker Overlapped Speech Recognition [27.35304346509647]
  We introduce speaker labels into an autoregressive transformer-based speech recognition model. We then propose a novel speaker mask branch to detection the speech segments of individual speakers. With the proposed model, we can perform both speech recognition and speaker diarization tasks simultaneously.
  arXiv  Detail & Related papers  (2023-12-18T06:29:53Z)
• End-to-End Single-Channel Speaker-Turn Aware Conversational Speech Translation [23.895122319920997]
  We tackle single-channel multi-speaker conversational ST with an end-to-end and multi-task training model. Speaker-Turn Aware Conversational Speech Translation combines automatic speech recognition, speech translation and speaker turn detection. We show that our model outperforms the reference systems on the multi-speaker condition, while attaining comparable performance on the single-speaker condition.
  arXiv  Detail & Related papers  (2023-11-01T17:55:09Z)
• TokenSplit: Using Discrete Speech Representations for Direct, Refined, and Transcript-Conditioned Speech Separation and Recognition [51.565319173790314]
  TokenSplit is a sequence-to-sequence encoder-decoder model that uses the Transformer architecture. We show that our model achieves excellent performance in terms of separation, both with or without transcript conditioning. We also measure the automatic speech recognition (ASR) performance and provide audio samples of speech synthesis to demonstrate the additional utility of our model.
  arXiv  Detail & Related papers  (2023-08-21T01:52:01Z)
• Streaming Speaker-Attributed ASR with Token-Level Speaker Embeddings [53.11450530896623]
  This paper presents a streaming speaker-attributed automatic speech recognition (SA-ASR) model that can recognize "who spoke what" Our model is based on token-level serialized output training (t-SOT) which was recently proposed to transcribe multi-talker speech in a streaming fashion. The proposed model achieves substantially better accuracy than a prior streaming model and shows comparable or sometimes even superior results to the state-of-the-art offline SA-ASR model.
  arXiv  Detail & Related papers  (2022-03-30T21:42:00Z)
• Investigating on Incorporating Pretrained and Learnable Speaker Representations for Multi-Speaker Multi-Style Text-to-Speech [54.75722224061665]
  In this work, we investigate different speaker representations and proposed to integrate pretrained and learnable speaker representations. The FastSpeech 2 model combined with both pretrained and learnable speaker representations shows great generalization ability on few-shot speakers.
  arXiv  Detail & Related papers  (2021-03-06T10:14:33Z)
• Joint Speaker Counting, Speech Recognition, and Speaker Identification for Overlapped Speech of Any Number of Speakers [38.3469744871394]
  We propose an end-to-end speaker-attributed automatic speech recognition model. It unifies speaker counting, speech recognition, and speaker identification on overlapped speech.
  arXiv  Detail & Related papers  (2020-06-19T02:05:18Z)
• Semi-supervised Learning for Multi-speaker Text-to-speech Synthesis Using Discrete Speech Representation [125.59372403631006]
  We propose a semi-supervised learning approach for multi-speaker text-to-speech (TTS) A multi-speaker TTS model can learn from the untranscribed audio via the proposed encoder-decoder framework with discrete speech representation. We found the model can benefit from the proposed semi-supervised learning approach even when part of the unpaired speech data is noisy.
  arXiv  Detail & Related papers  (2020-05-16T15:47:11Z)
• Voice Separation with an Unknown Number of Multiple Speakers [113.91855071999298]
  We present a new method for separating a mixed audio sequence, in which multiple voices speak simultaneously. The new method employs gated neural networks that are trained to separate the voices at multiple processing steps, while maintaining the speaker in each output channel fixed.
  arXiv  Detail & Related papers  (2020-02-29T20:02:54Z)
• Unsupervised Audiovisual Synthesis via Exemplar Autoencoders [59.13989658692953]
  We present an unsupervised approach that converts the input speech of any individual into audiovisual streams of potentially-infinitely many output speakers. We use Exemplar Autoencoders to learn the voice, stylistic prosody, and visual appearance of a specific target speech exemplar.
  arXiv  Detail & Related papers  (2020-01-13T18:56:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.