Wav2code: Restore Clean Speech Representations via Codebook Lookup for Noise-Robust ASR

• URL: http://arxiv.org/abs/2304.04974v3
• Date: Thu, 18 Apr 2024 06:39:57 GMT
• Title: Wav2code: Restore Clean Speech Representations via Codebook Lookup for Noise-Robust ASR
• Authors: Yuchen Hu, Chen Chen, Qiushi Zhu, Eng Siong Chng,
• Abstract summary: We propose a self-supervised framework named Wav2code to implement a feature-level SE with reduced distortions for noise-robust ASR. During finetuning, we propose a Transformer-based code predictor to accurately predict clean codes by modeling the global dependency of input noisy representations. Experiments on both synthetic and real noisy datasets demonstrate that Wav2code can solve the speech distortion and improve ASR performance under various noisy conditions.
• Score: 35.710735895190844
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Automatic speech recognition (ASR) has gained remarkable successes thanks to recent advances of deep learning, but it usually degrades significantly under real-world noisy conditions. Recent works introduce speech enhancement (SE) as front-end to improve speech quality, which is proved effective but may not be optimal for downstream ASR due to speech distortion problem. Based on that, latest works combine SE and currently popular self-supervised learning (SSL) to alleviate distortion and improve noise robustness. Despite the effectiveness, the speech distortion caused by conventional SE still cannot be cleared out. In this paper, we propose a self-supervised framework named Wav2code to implement a feature-level SE with reduced distortions for noise-robust ASR. First, in pre-training stage the clean speech representations from SSL model are sent to lookup a discrete codebook via nearest-neighbor feature matching, the resulted code sequence are then exploited to reconstruct the original clean representations, in order to store them in codebook as prior. Second, during finetuning we propose a Transformer-based code predictor to accurately predict clean codes by modeling the global dependency of input noisy representations, which enables discovery and restoration of high-quality clean representations with reduced distortions. Furthermore, we propose an interactive feature fusion network to combine original noisy and the restored clean representations to consider both fidelity and quality, resulting in more informative features for downstream ASR. Finally, experiments on both synthetic and real noisy datasets demonstrate that Wav2code can solve the speech distortion and improve ASR performance under various noisy conditions, resulting in stronger robustness.

Related papers

• TRNet: Two-level Refinement Network leveraging Speech Enhancement for Noise Robust Speech Emotion Recognition [29.756961194844717]
  The proposed TRNet substantially promotes the robustness of the proposed system in both matched and unmatched noisy environments. Results validate that the proposed system substantially promotes the robustness of the proposed system in both matched and unmatched noisy environments.
  arXiv  Detail & Related papers  (2024-04-19T16:09:17Z)
• Bring the Noise: Introducing Noise Robustness to Pretrained Automatic Speech Recognition [13.53738829631595]
  We propose a novel method to extract the denoising capabilities, that can be applied to any encoder-decoder architecture. We train our pre-processor on the Noisy Speech Database (NSD) to reconstruct denoised spectrograms from noisy inputs. We show that the Cleancoder is able to filter noise from speech and that it improves the total Word Error Rate (WER) of the downstream model in noisy conditions.
  arXiv  Detail & Related papers  (2023-09-05T11:34:21Z)
• NLIP: Noise-robust Language-Image Pre-training [95.13287735264937]
  We propose a principled Noise-robust Language-Image Pre-training framework (NLIP) to stabilize pre-training via two schemes: noise-harmonization and noise-completion. Our NLIP can alleviate the common noise effects during image-text pre-training in a more efficient way.
  arXiv  Detail & Related papers  (2022-12-14T08:19:30Z)
• Self-Supervised Learning for Speech Enhancement through Synthesis [5.924928860260821]
  We propose a denoising vocoder (DeVo) approach, where a vocoder accepts noisy representations and learns to directly synthesize clean speech. We demonstrate a causal version capable of running on streaming audio with 10ms latency and minimal performance degradation.
  arXiv  Detail & Related papers  (2022-11-04T16:06:56Z)
• Discretization and Re-synthesis: an alternative method to solve the Cocktail Party Problem [65.25725367771075]
  This study demonstrates, for the first time, that the synthesis-based approach can also perform well on this problem. Specifically, we propose a novel speech separation/enhancement model based on the recognition of discrete symbols. By utilizing the synthesis model with the input of discrete symbols, after the prediction of discrete symbol sequence, each target speech could be re-synthesized.
  arXiv  Detail & Related papers  (2021-12-17T08:35:40Z)
• Improving Noise Robustness of Contrastive Speech Representation Learning with Speech Reconstruction [109.44933866397123]
  Noise robustness is essential for deploying automatic speech recognition systems in real-world environments. We employ a noise-robust representation learned by a refined self-supervised framework for noisy speech recognition. We achieve comparable performance to the best supervised approach reported with only 16% of labeled data.
  arXiv  Detail & Related papers  (2021-10-28T20:39:02Z)
• Wav2vec-Switch: Contrastive Learning from Original-noisy Speech Pairs for Robust Speech Recognition [52.71604809100364]
  We propose wav2vec-Switch, a method to encode noise robustness into contextualized representations of speech. Specifically, we feed original-noisy speech pairs simultaneously into the wav2vec 2.0 network. In addition to the existing contrastive learning task, we switch the quantized representations of the original and noisy speech as additional prediction targets.
  arXiv  Detail & Related papers  (2021-10-11T00:08:48Z)
• Variational Autoencoder for Speech Enhancement with a Noise-Aware Encoder [30.318947721658862]
  We propose to include noise information in the training phase by using a noise-aware encoder trained on noisy-clean speech pairs. We show that our proposed noise-aware VAE outperforms the standard VAE in terms of overall distortion without increasing the number of model parameters.
  arXiv  Detail & Related papers  (2021-02-17T11:40:42Z)
• Dual Adversarial Network: Toward Real-world Noise Removal and Noise Generation [52.75909685172843]
  Real-world image noise removal is a long-standing yet very challenging task in computer vision. We propose a novel unified framework to deal with the noise removal and noise generation tasks. Our method learns the joint distribution of the clean-noisy image pairs.
  arXiv  Detail & Related papers  (2020-07-12T09:16:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.