Advancing Topic Segmentation of Broadcasted Speech with Multilingual Semantic Embeddings

• URL: http://arxiv.org/abs/2409.06222v1
• Date: Tue, 10 Sep 2024 05:24:36 GMT
• Title: Advancing Topic Segmentation of Broadcasted Speech with Multilingual Semantic Embeddings
• Authors: Sakshi Deo Shukla, Pavel Denisov, Tugtekin Turan,
• Abstract summary: We introduce an end-to-end scheme for topic segmentation using semantic speech encoders. We propose a new benchmark for spoken news topic segmentation by utilizing a dataset featuring 1000 hours of publicly available recordings. Our results demonstrate that while the traditional pipeline approach achieves a state-of-the-art $P_k$ score of 0.2431 for English, our end-to-end model delivers a competitive $P_k$ score of 0.2564.
• Score: 2.615008111842321
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Recent advancements in speech-based topic segmentation have highlighted the potential of pretrained speech encoders to capture semantic representations directly from speech. Traditionally, topic segmentation has relied on a pipeline approach in which transcripts of the automatic speech recognition systems are generated, followed by text-based segmentation algorithms. In this paper, we introduce an end-to-end scheme that bypasses this conventional two-step process by directly employing semantic speech encoders for segmentation. Focused on the broadcasted news domain, which poses unique challenges due to the diversity of speakers and topics within single recordings, we address the challenge of accessing topic change points efficiently in an end-to-end manner. Furthermore, we propose a new benchmark for spoken news topic segmentation by utilizing a dataset featuring approximately 1000 hours of publicly available recordings across six European languages and including an evaluation set in Hindi to test the model's cross-domain performance in a cross-lingual, zero-shot scenario. This setup reflects real-world diversity and the need for models adapting to various linguistic settings. Our results demonstrate that while the traditional pipeline approach achieves a state-of-the-art $P_k$ score of 0.2431 for English, our end-to-end model delivers a competitive $P_k$ score of 0.2564. When trained multilingually, these scores further improve to 0.1988 and 0.2370, respectively. To support further research, we release our model along with data preparation scripts, facilitating open research on multilingual spoken news topic segmentation.

Related papers

• TransVIP: Speech to Speech Translation System with Voice and Isochrony Preservation [97.54885207518946]
  We introduce a novel model framework TransVIP that leverages diverse datasets in a cascade fashion. We propose two separated encoders to preserve the speaker's voice characteristics and isochrony from the source speech during the translation process. Our experiments on the French-English language pair demonstrate that our model outperforms the current state-of-the-art speech-to-speech translation model.
  arXiv  Detail & Related papers  (2024-05-28T04:11:37Z)
• Few-Shot Spoken Language Understanding via Joint Speech-Text Models [18.193191170754744]
  Recent work on speech representation models jointly pre-trained with text has demonstrated the potential of improving speech representations. We leverage such shared representations to address the persistent challenge of limited data availability in spoken language understanding tasks. By employing a pre-trained speech-text model, we find that models fine-tuned on text can be effectively transferred to speech testing data.
  arXiv  Detail & Related papers  (2023-10-09T17:59:21Z)
• Long-Form End-to-End Speech Translation via Latent Alignment Segmentation [6.153530338207679]
  Current simultaneous speech translation models can process audio only up to a few seconds long. We propose a novel segmentation approach for a low-latency end-to-end speech translation. We show that the proposed approach achieves state-of-the-art quality at no additional computational cost.
  arXiv  Detail & Related papers  (2023-09-20T15:10:12Z)
• Cross-lingual Knowledge Distillation via Flow-based Voice Conversion for Robust Polyglot Text-To-Speech [6.243356997302935]
  We introduce a framework for cross-lingual speech synthesis, which involves an upstream Voice Conversion (VC) model and a downstream Text-To-Speech (TTS) model. In the first two stages, we use a VC model to convert utterances in the target locale to the voice of the target speaker. In the third stage, the converted data is combined with the linguistic features and durations from recordings in the target language, which are then used to train a single-speaker acoustic model.
  arXiv  Detail & Related papers  (2023-09-15T09:03:14Z)
• ComSL: A Composite Speech-Language Model for End-to-End Speech-to-Text Translation [79.66359274050885]
  We present ComSL, a speech-language model built atop a composite architecture of public pretrained speech-only and language-only models. Our approach has demonstrated effectiveness in end-to-end speech-to-text translation tasks.
  arXiv  Detail & Related papers  (2023-05-24T07:42:15Z)
• Google USM: Scaling Automatic Speech Recognition Beyond 100 Languages [76.95115818308918]
  We introduce the Universal Speech Model (USM), a single large model that performs automatic speech recognition (ASR) across 100+ languages. This is achieved by pre-training the encoder of the model on a large unlabeled multilingual dataset of 12 million (M) hours spanning over 300 languages. We use multilingual pre-training with random-projection quantization and speech-text modality matching to achieve state-of-the-art performance on downstream multilingual ASR and speech-to-text translation tasks.
  arXiv  Detail & Related papers  (2023-03-02T07:47:18Z)
• Pretraining Approaches for Spoken Language Recognition: TalTech Submission to the OLR 2021 Challenge [0.0]
  The paper is based on our submission to the Oriental Language Recognition 2021 Challenge. For the constrained track, we first trained a Conformer-based encoder-decoder model for multilingual automatic speech recognition. For the unconstrained task, we relied on both externally available pretrained models as well as external data.
  arXiv  Detail & Related papers  (2022-05-14T15:17:08Z)
• Direct speech-to-speech translation with discrete units [64.19830539866072]
  We present a direct speech-to-speech translation (S2ST) model that translates speech from one language to speech in another language without relying on intermediate text generation. We propose to predict the self-supervised discrete representations learned from an unlabeled speech corpus instead. When target text transcripts are available, we design a multitask learning framework with joint speech and text training that enables the model to generate dual mode output (speech and text) simultaneously in the same inference pass.
  arXiv  Detail & Related papers  (2021-07-12T17:40:43Z)
• Bridging the Modality Gap for Speech-to-Text Translation [57.47099674461832]
  End-to-end speech translation aims to translate speech in one language into text in another language via an end-to-end way. Most existing methods employ an encoder-decoder structure with a single encoder to learn acoustic representation and semantic information simultaneously. We propose a Speech-to-Text Adaptation for Speech Translation model which aims to improve the end-to-end model performance by bridging the modality gap between speech and text.
  arXiv  Detail & Related papers  (2020-10-28T12:33:04Z)
• Multitask Training with Text Data for End-to-End Speech Recognition [45.35605825009208]
  We propose a multitask training method for attention-based end-to-end speech recognition models. We regularize the decoder in a listen, attend, and spell model by multitask training it on both audio-text and text-only data.
  arXiv  Detail & Related papers  (2020-10-27T14:29:28Z)
• Cross-lingual Spoken Language Understanding with Regularized Representation Alignment [71.53159402053392]
  We propose a regularization approach to align word-level and sentence-level representations across languages without any external resource. Experiments on the cross-lingual spoken language understanding task show that our model outperforms current state-of-the-art methods in both few-shot and zero-shot scenarios.
  arXiv  Detail & Related papers  (2020-09-30T08:56:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.