Pretraining End-to-End Keyword Search with Automatically Discovered Acoustic Units

• URL: http://arxiv.org/abs/2407.04652v1
• Date: Fri, 5 Jul 2024 17:07:58 GMT
• Title: Pretraining End-to-End Keyword Search with Automatically Discovered Acoustic Units
• Authors: Bolaji Yusuf, Jan "Honza" Černocký, Murat Saraçlar,
• Abstract summary: We propose a method for pretraining E2E KWS systems with untranscribed data. We show that finetuning such a model significantly outperforms a model trained from scratch.
• Score: 8.86336076082867
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: End-to-end (E2E) keyword search (KWS) has emerged as an alternative and complimentary approach to conventional keyword search which depends on the output of automatic speech recognition (ASR) systems. While E2E methods greatly simplify the KWS pipeline, they generally have worse performance than their ASR-based counterparts, which can benefit from pretraining with untranscribed data. In this work, we propose a method for pretraining E2E KWS systems with untranscribed data, which involves using acoustic unit discovery (AUD) to obtain discrete units for untranscribed data and then learning to locate sequences of such units in the speech. We conduct experiments across languages and AUD systems: we show that finetuning such a model significantly outperforms a model trained from scratch, and the performance improvements are generally correlated with the quality of the AUD system used for pretraining.

Related papers

• Written Term Detection Improves Spoken Term Detection [9.961529254621432]
  We propose a multitask training objective which allows unpaired text to be integrated into E2E KWS without complicating indexing and search. In addition to training an E2E KWS model to retrieve text queries from spoken documents, we jointly train it to retrieve text queries from masked written documents. We show that this approach can effectively leverage unpaired text for KWS, with significant improvements in search performance across a wide variety of languages.
  arXiv  Detail & Related papers  (2024-07-05T15:50:47Z)
• D4AM: A General Denoising Framework for Downstream Acoustic Models [45.04967351760919]
  Speech enhancement (SE) can be used as a front-end strategy to aid automatic speech recognition (ASR) systems. Existing training objectives of SE methods are not fully effective at integrating speech-text and noisy-clean paired data for training toward unseen ASR systems. We propose a general denoising framework, D4AM, for various downstream acoustic models.
  arXiv  Detail & Related papers  (2023-11-28T08:27:27Z)
• UnitY: Two-pass Direct Speech-to-speech Translation with Discrete Units [64.61596752343837]
  We present a novel two-pass direct S2ST architecture, UnitY, which first generates textual representations and predicts discrete acoustic units. We enhance the model performance by subword prediction in the first-pass decoder. We show that the proposed methods boost the performance even when predicting spectrogram in the second pass.
  arXiv  Detail & Related papers  (2022-12-15T18:58:28Z)
• Fully Automated End-to-End Fake Audio Detection [57.78459588263812]
  This paper proposes a fully automated end-toend fake audio detection method. We first use wav2vec pre-trained model to obtain a high-level representation of the speech. For the network structure, we use a modified version of the differentiable architecture search (DARTS) named light-DARTS.
  arXiv  Detail & Related papers  (2022-08-20T06:46:55Z)
• Learning Phone Recognition from Unpaired Audio and Phone Sequences Based on Generative Adversarial Network [58.82343017711883]
  This paper investigates how to learn directly from unpaired phone sequences and speech utterances. GAN training is adopted in the first stage to find the mapping relationship between unpaired speech and phone sequence. In the second stage, another HMM model is introduced to train from the generator's output, which boosts the performance.
  arXiv  Detail & Related papers  (2022-07-29T09:29:28Z)
• Revisiting End-to-End Speech-to-Text Translation From Scratch [48.203394370942505]
  End-to-end (E2E) speech-to-text translation (ST) often depends on pretraining its encoder and/or decoder using source transcripts via speech recognition or text translation tasks. In this paper, we explore the extent to which the quality of E2E ST trained on speech-translation pairs alone can be improved.
  arXiv  Detail & Related papers  (2022-06-09T15:39:19Z)
• Wav2Seq: Pre-training Speech-to-Text Encoder-Decoder Models Using Pseudo Languages [58.43299730989809]
  We introduce Wav2Seq, the first self-supervised approach to pre-train both parts of encoder-decoder models for speech data. We induce a pseudo language as a compact discrete representation, and formulate a self-supervised pseudo speech recognition task. This process stands on its own, or can be applied as low-cost second-stage pre-training.
  arXiv  Detail & Related papers  (2022-05-02T17:59:02Z)
• Enhanced Direct Speech-to-Speech Translation Using Self-supervised Pre-training and Data Augmentation [76.13334392868208]
  Direct speech-to-speech translation (S2ST) models suffer from data scarcity issues. In this work, we explore self-supervised pre-training with unlabeled speech data and data augmentation to tackle this issue.
  arXiv  Detail & Related papers  (2022-04-06T17:59:22Z)
• A Complementary Joint Training Approach Using Unpaired Speech and Text for Low-Resource Automatic Speech Recognition [25.473191378558138]
  We leverage unpaired data to train a general sequence-to-sequence model. Inspired by the complementarity of speech-PseudoLabel pair and SynthesizedAudio-text pair, we propose a complementary joint training(CJT) method.
  arXiv  Detail & Related papers  (2022-04-05T07:02:53Z)
• An Exploration of Self-Supervised Pretrained Representations for End-to-End Speech Recognition [98.70304981174748]
  We focus on the general applications of pretrained speech representations, on advanced end-to-end automatic speech recognition (E2E-ASR) models. We select several pretrained speech representations and present the experimental results on various open-source and publicly available corpora for E2E-ASR.
  arXiv  Detail & Related papers  (2021-10-09T15:06:09Z)
• Active Learning for Sound Event Detection [18.750572243562576]
  This paper proposes an active learning system for sound event detection (SED) It aims at maximizing the accuracy of a learned SED model with limited annotation effort. Remarkably, the required annotation effort can be greatly reduced on the dataset where target sound events are rare.
  arXiv  Detail & Related papers  (2020-02-12T14:46:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.