LASER: Learning by Aligning Self-supervised Representations of Speech for Improving Content-related Tasks

• URL: http://arxiv.org/abs/2406.09153v1
• Date: Thu, 13 Jun 2024 14:17:47 GMT
• Title: LASER: Learning by Aligning Self-supervised Representations of Speech for Improving Content-related Tasks
• Authors: Amit Meghanani, Thomas Hain,
• Abstract summary: A cost-effective self-supervised fine-tuning (SSFT) method named "LASER: Learning by Aligning Self-supervised Representations" is presented. Experiments are conducted with HuBERT and WavLM models and evaluated on the SUPERB benchmark for two content-related tasks: automatic speech recognition (ASR) and phoneme recognition (PR) A relative improvement of 3.7% and 8.2% for HuBERT, and 4.1% and 11.7% for WavLM are observed, for the ASR and PR tasks respectively, with only 3 hours of fine-tuning on a single GPU.
• Score: 19.94790551312789
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Self-supervised learning (SSL)-based speech models are extensively used for full-stack speech processing. However, it has been observed that improving SSL-based speech representations using unlabeled speech for content-related tasks is challenging and computationally expensive. Recent attempts have been made to address this issue with cost-effective self-supervised fine-tuning (SSFT) approaches. Continuing in this direction, a cost-effective SSFT method named "LASER: Learning by Aligning Self-supervised Representations" is presented. LASER is based on the soft-DTW alignment loss with temporal regularisation term. Experiments are conducted with HuBERT and WavLM models and evaluated on the SUPERB benchmark for two content-related tasks: automatic speech recognition (ASR) and phoneme recognition (PR). A relative improvement of 3.7% and 8.2% for HuBERT, and 4.1% and 11.7% for WavLM are observed, for the ASR and PR tasks respectively, with only < 3 hours of fine-tuning on a single GPU.

Related papers

• Large Language Models Are Strong Audio-Visual Speech Recognition Learners [53.142635674428874]
  Multimodal large language models (MLLMs) have recently become a focal point of research due to their formidable multimodal understanding capabilities. We propose Llama-AVSR, a new MLLM with strong audio-visual speech recognition capabilities. We evaluate our proposed approach on LRS3, the largest public AVSR benchmark, and we achieve new state-of-the-art results for the tasks of ASR and AVSR with a WER of 0.81% and 0.77%, respectively.
  arXiv  Detail & Related papers  (2024-09-18T21:17:27Z)
• SCORE: Self-supervised Correspondence Fine-tuning for Improved Content Representations [23.56580783289533]
  This work presents a cost-effective SSFT method named Self-supervised Correspondence (SCORE) fine-tuning to adapt the SSL speech representations for content-related tasks. SCORE outperforms the vanilla HuBERT on SUPERB benchmark with only a few hours of fine-tuning ( 5 hrs) on a single GPU for automatic speech recognition, phoneme recognition, and query-by-example tasks.
  arXiv  Detail & Related papers  (2024-03-10T16:57:51Z)
• ML-LMCL: Mutual Learning and Large-Margin Contrastive Learning for Improving ASR Robustness in Spoken Language Understanding [55.39105863825107]
  We propose Mutual Learning and Large-Margin Contrastive Learning (ML-LMCL) to improve automatic speech recognition (ASR) robustness. In fine-tuning, we apply mutual learning and train two SLU models on the manual transcripts and the ASR transcripts, respectively. Experiments on three datasets show that ML-LMCL outperforms existing models and achieves new state-of-the-art performance.
  arXiv  Detail & Related papers  (2023-11-19T16:53:35Z)
• Self-supervised Neural Factor Analysis for Disentangling Utterance-level Speech Representations [30.293081541301746]
  Self-supervised learning (SSL) speech models such as wav2vec and HuBERT have demonstrated state-of-the-art performance on automatic speech recognition. We argue that the problem is caused by the lack of disentangled representations and an utterance-level learning objective. Our models outperform the current best model, WavLM, on all utterance-level non-semantic tasks on the SUPERB benchmark with only 20% of labeled data.
  arXiv  Detail & Related papers  (2023-05-14T08:26:24Z)
• Supervision-Guided Codebooks for Masked Prediction in Speech Pre-training [102.14558233502514]
  Masked prediction pre-training has seen remarkable progress in self-supervised learning (SSL) for speech recognition. We propose two supervision-guided codebook generation approaches to improve automatic speech recognition (ASR) performance.
  arXiv  Detail & Related papers  (2022-06-21T06:08:30Z)
• A Fine-tuned Wav2vec 2.0/HuBERT Benchmark For Speech Emotion Recognition, Speaker Verification and Spoken Language Understanding [0.9023847175654603]
  We explore partial fine-tuning and entire fine-tuning on wav2vec 2.0 and HuBERT pre-trained models for three non-ASR speech tasks. With simple down-stream frameworks, the best scores reach 79.58% weighted accuracy for Speech Emotion Recognition on IEMOCAP, 2.36% equal error rate for Speaker Verification on VoxCeleb1, 87.51% accuracy for Intent Classification and 75.32% F1 for Slot Filling on SLURP.
  arXiv  Detail & Related papers  (2021-11-04T10:39:06Z)
• WavLM: Large-Scale Self-Supervised Pre-Training for Full Stack Speech Processing [102.45426364965887]
  We propose a new pre-trained model, WavLM, to solve full-stack downstream speech tasks. WavLM is built based on the HuBERT framework, with an emphasis on both spoken content modeling and speaker identity preservation. We scale up the training dataset from 60k hours to 94k hours of public audio data, and optimize its training procedure for better representation extraction.
  arXiv  Detail & Related papers  (2021-10-26T17:55:19Z)
• UniSpeech-SAT: Universal Speech Representation Learning with Speaker Aware Pre-Training [72.004873454347]
  Two methods are introduced for enhancing the unsupervised speaker information extraction. Experiment results on SUPERB benchmark show that the proposed system achieves state-of-the-art performance. We scale up training dataset to 94 thousand hours public audio data and achieve further performance improvement.
  arXiv  Detail & Related papers  (2021-10-12T05:43:30Z)
• Task-aware Warping Factors in Mask-based Speech Enhancement [31.913984833849753]
  We propose the use of two task-aware warping factors in mask-based speech enhancement (SE) One controls the balance between speech-maintenance and noise-removal in training phases, while the other controls SE power applied to specific downstream tasks. It is easy to apply the proposed dual-warping factors approach to any mask-based SE method.
  arXiv  Detail & Related papers  (2021-08-27T05:57:37Z)
• LeBenchmark: A Reproducible Framework for Assessing Self-Supervised Representation Learning from Speech [63.84741259993937]
  Self-Supervised Learning (SSL) using huge unlabeled data has been successfully explored for image and natural language processing. Recent works also investigated SSL from speech. We propose LeBenchmark: a reproducible framework for assessing SSL from speech.
  arXiv  Detail & Related papers  (2021-04-23T08:27:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.