Self-supervised Neural Factor Analysis for Disentangling Utterance-level Speech Representations

• URL: http://arxiv.org/abs/2305.08099v3
• Date: Wed, 4 Oct 2023 12:15:56 GMT
• Title: Self-supervised Neural Factor Analysis for Disentangling Utterance-level Speech Representations
• Authors: Weiwei Lin, Chenhang He, Man-Wai Mak, Youzhi Tu
• Abstract summary: Self-supervised learning (SSL) speech models such as wav2vec and HuBERT have demonstrated state-of-the-art performance on automatic speech recognition. We argue that the problem is caused by the lack of disentangled representations and an utterance-level learning objective. Our models outperform the current best model, WavLM, on all utterance-level non-semantic tasks on the SUPERB benchmark with only 20% of labeled data.
• Score: 30.293081541301746
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-supervised learning (SSL) speech models such as wav2vec and HuBERT have demonstrated state-of-the-art performance on automatic speech recognition (ASR) and proved to be extremely useful in low label-resource settings. However, the success of SSL models has yet to transfer to utterance-level tasks such as speaker, emotion, and language recognition, which still require supervised fine-tuning of the SSL models to obtain good performance. We argue that the problem is caused by the lack of disentangled representations and an utterance-level learning objective for these tasks. Inspired by how HuBERT uses clustering to discover hidden acoustic units, we formulate a factor analysis (FA) model that uses the discovered hidden acoustic units to align the SSL features. The underlying utterance-level representations are disentangled from the content of speech using probabilistic inference on the aligned features. Furthermore, the variational lower bound derived from the FA model provides an utterance-level objective, allowing error gradients to be backpropagated to the Transformer layers to learn highly discriminative acoustic units. When used in conjunction with HuBERT's masked prediction training, our models outperform the current best model, WavLM, on all utterance-level non-semantic tasks on the SUPERB benchmark with only 20% of labeled data.

Related papers

• A Closer Look at Wav2Vec2 Embeddings for On-Device Single-Channel Speech Enhancement [16.900731393703648]
  Self-supervised learned models have been found to be very effective for certain speech tasks. In this paper, we investigate the uses of SSL representations for single-channel speech enhancement in challenging conditions.
  arXiv  Detail & Related papers  (2024-03-03T02:05:17Z)
• Pushing the Limits of Unsupervised Unit Discovery for SSL Speech Representation [12.506633315768832]
  HuBERT is a successful example that utilizes offline clustering to convert speech features into discrete units for a masked language modeling pretext task. We present an unsupervised method to improve SSL targets. Two models are proposed, MonoBERT and PolyBERT, which leverage context-independent and context-dependent phoneme-based units for pre-training.
  arXiv  Detail & Related papers  (2023-06-15T07:45:12Z)
• Self-supervised models of audio effectively explain human cortical responses to speech [71.57870452667369]
  We capitalize on the progress of self-supervised speech representation learning to create new state-of-the-art models of the human auditory system. We show that these results show that self-supervised models effectively capture the hierarchy of information relevant to different stages of speech processing in human cortex.
  arXiv  Detail & Related papers  (2022-05-27T22:04:02Z)
• Why does Self-Supervised Learning for Speech Recognition Benefit Speaker Recognition? [86.53044183309824]
  We study which factor leads to the success of self-supervised learning on speaker-related tasks. Our empirical results on the Voxceleb-1 dataset suggest that the benefit of SSL to SV task is from a combination of mask speech prediction loss, data scale, and model size.
  arXiv  Detail & Related papers  (2022-04-27T08:35:57Z)
• Automatic Pronunciation Assessment using Self-Supervised Speech Representation Learning [13.391307807956673]
  We propose a novel automatic pronunciation assessment method based on self-supervised learning (SSL) models. First, the proposed method fine-tunes the pre-trained SSL models with connectionist temporal classification to adapt the English pronunciation of English-as-a-second-language learners. We show that the proposed SSL model-based methods outperform the baselines, in terms of the Pearson correlation coefficient, on datasets of Korean ESL learner children and Speechocean762.
  arXiv  Detail & Related papers  (2022-04-08T06:13:55Z)
• An Exploration of Prompt Tuning on Generative Spoken Language Model for Speech Processing Tasks [112.1942546460814]
  We report the first exploration of the prompt tuning paradigm for speech processing tasks based on Generative Spoken Language Model (GSLM) Experiment results show that the prompt tuning technique achieves competitive performance in speech classification tasks with fewer trainable parameters than fine-tuning specialized downstream models.
  arXiv  Detail & Related papers  (2022-03-31T03:26:55Z)
• Self-Supervised Learning for speech recognition with Intermediate layer supervision [52.93758711230248]
  We propose Intermediate Layer Supervision for Self-Supervised Learning (ILS-SSL) ILS-SSL forces the model to concentrate on content information as much as possible by adding an additional SSL loss on the intermediate layers. Experiments on LibriSpeech test-other set show that our method outperforms HuBERT significantly.
  arXiv  Detail & Related papers  (2021-12-16T10:45:05Z)
• UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training [72.004873454347]
  Two methods are introduced for enhancing the unsupervised speaker information extraction. Experiment results on SUPERB benchmark show that the proposed system achieves state-of-the-art performance. We scale up training dataset to 94 thousand hours public audio data and achieve further performance improvement.
  arXiv  Detail & Related papers  (2021-10-12T05:43:30Z)
• Preliminary study on using vector quantization latent spaces for TTS/VC systems with consistent performance [55.10864476206503]
  We investigate the use of quantized vectors to model the latent linguistic embedding. By enforcing different policies over the latent spaces in the training, we are able to obtain a latent linguistic embedding. Our experiments show that the voice cloning system built with vector quantization has only a small degradation in terms of perceptive evaluations.
  arXiv  Detail & Related papers  (2021-06-25T07:51:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.