Feature Normalization for Fine-tuning Self-Supervised Models in Speech Enhancement

• URL: http://arxiv.org/abs/2306.08406v1
• Date: Wed, 14 Jun 2023 10:03:33 GMT
• Title: Feature Normalization for Fine-tuning Self-Supervised Models in Speech Enhancement
• Authors: Hejung Yang, Hong-Goo Kang
• Abstract summary: Large, pre-trained representation models trained using self-supervised learning have gained popularity in various fields of machine learning. In this paper, we investigate the feasibility of using pre-trained speech representation models for a downstream speech enhancement task. Our proposed method enables significant improvements in speech quality compared to baselines when combined with various types of pre-trained speech models.
• Score: 19.632358491434697
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large, pre-trained representation models trained using self-supervised learning have gained popularity in various fields of machine learning because they are able to extract high-quality salient features from input data. As such, they have been frequently used as base networks for various pattern classification tasks such as speech recognition. However, not much research has been conducted on applying these types of models to the field of speech signal generation. In this paper, we investigate the feasibility of using pre-trained speech representation models for a downstream speech enhancement task. To alleviate mismatches between the input features of the pre-trained model and the target enhancement model, we adopt a novel feature normalization technique to smoothly link these modules together. Our proposed method enables significant improvements in speech quality compared to baselines when combined with various types of pre-trained speech models.

Related papers

• Transferable Post-training via Inverse Value Learning [83.75002867411263]
  We propose modeling changes at the logits level during post-training using a separate neural network (i.e., the value network) After training this network on a small base model using demonstrations, this network can be seamlessly integrated with other pre-trained models during inference. We demonstrate that the resulting value network has broad transferability across pre-trained models of different parameter sizes.
  arXiv  Detail & Related papers  (2024-10-28T13:48:43Z)
• Ensembling Finetuned Language Models for Text Classification [55.15643209328513]
  Finetuning is a common practice across different communities to adapt pretrained models to particular tasks. ensembles of neural networks are typically used to boost performance and provide reliable uncertainty estimates. We present a metadataset with predictions from five large finetuned models on six datasets and report results of different ensembling strategies.
  arXiv  Detail & Related papers  (2024-10-25T09:15:54Z)
• SpeechGPT-Gen: Scaling Chain-of-Information Speech Generation [56.913182262166316]
  Chain-of-Information Generation (CoIG) is a method for decoupling semantic and perceptual information in large-scale speech generation. SpeechGPT-Gen is efficient in semantic and perceptual information modeling. It markedly excels in zero-shot text-to-speech, zero-shot voice conversion, and speech-to-speech dialogue.
  arXiv  Detail & Related papers  (2024-01-24T15:25:01Z)
• Generative Pre-training for Speech with Flow Matching [81.59952572752248]
  We pre-trained a generative model, named SpeechFlow, on 60k hours of untranscribed speech with Flow Matching and masked conditions. Experiment results show the pre-trained generative model can be fine-tuned with task-specific data to match or surpass existing expert models on speech enhancement, separation, and synthesis.
  arXiv  Detail & Related papers  (2023-10-25T03:40:50Z)
• Augmentation Invariant Discrete Representation for Generative Spoken Language Modeling [41.733860809136196]
  We propose an effective and efficient method to learn robust discrete speech representation for generative spoken language modeling. The proposed approach is based on applying a set of signal transformations to the speech signal and optimizing the model using an iterative pseudo-labeling scheme. We additionally evaluate our method on the speech-to-speech translation task, considering Spanish-English and French-English translations, and show the proposed approach outperforms the evaluated baselines.
  arXiv  Detail & Related papers  (2022-09-30T14:15:03Z)
• An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks [112.1942546460814]
  We report the first exploration of the prompt tuning paradigm for speech processing tasks based on Generative Spoken Language Model (GSLM) Experiment results show that the prompt tuning technique achieves competitive performance in speech classification tasks with fewer trainable parameters than fine-tuning specialized downstream models.
  arXiv  Detail & Related papers  (2022-03-31T03:26:55Z)
• Layer-wise Analysis of a Self-supervised Speech Representation Model [26.727775920272205]
  Self-supervised learning approaches have been successful for pre-training speech representation models. Not much has been studied about the type or extent of information encoded in the pre-trained representations themselves.
  arXiv  Detail & Related papers  (2021-07-10T02:13:25Z)
• Self-Supervised Learning for Personalized Speech Enhancement [25.05285328404576]
  Speech enhancement systems can show improved performance by adapting the model towards a single test-time speaker. Test-time user might only provide a small amount of noise-free speech data, likely insufficient for traditional fully-supervised learning. We propose self-supervised methods that are designed specifically to learn personalized and discriminative features from abundant in-the-wild noisy, but still personal speech recordings.
  arXiv  Detail & Related papers  (2021-04-05T17:12:51Z)
• An Investigation of End-to-End Models for Robust Speech Recognition [20.998349142078805]
  We present a comparison of speech enhancement-based techniques and three different model-based adaptation techniques for robust automatic speech recognition. While adversarial learning is the best-performing technique on certain noise types, it comes at the cost of degrading clean speech WER. On other relatively stationary noise types, a new speech enhancement technique outperformed all the model-based adaptation techniques.
  arXiv  Detail & Related papers  (2021-02-11T19:47:13Z)
• TERA: Self-Supervised Learning of Transformer Encoder Representation for Speech [63.03318307254081]
  TERA stands for Transformer Representations from Alteration. We use alteration along three axes to pre-train Transformers on a large amount of unlabeled speech. TERA can be used for speech representations extraction or fine-tuning with downstream models.
  arXiv  Detail & Related papers  (2020-07-12T16:19:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.