EH-MAM: Easy-to-Hard Masked Acoustic Modeling for Self-Supervised Speech Representation Learning

• URL: http://arxiv.org/abs/2410.13179v1
• Date: Thu, 17 Oct 2024 02:59:22 GMT
• Title: EH-MAM: Easy-to-Hard Masked Acoustic Modeling for Self-Supervised Speech Representation Learning
• Authors: Ashish Seth, Ramaneswaran Selvakumar, S Sakshi, Sonal Kumar, Sreyan Ghosh, Dinesh Manocha,
• Abstract summary: EH-MAM (Easy-to-Hard adaptive Masked Acoustic Modeling) is a novel self-supervised learning approach for speech representation learning. We introduce a novel selective and adaptive masking strategy for Masked Acoustic Modeling (MAM) EH-MAM outperforms several state-of-the-art baselines across various low-resource speech recognition and SUPERB benchmarks by 5%-10%.
• Score: 46.66166658067071
• License:
• Abstract: In this paper, we present EH-MAM (Easy-to-Hard adaptive Masked Acoustic Modeling), a novel self-supervised learning approach for speech representation learning. In contrast to the prior methods that use random masking schemes for Masked Acoustic Modeling (MAM), we introduce a novel selective and adaptive masking strategy. Specifically, during SSL training, we progressively introduce harder regions to the model for reconstruction. Our approach automatically selects hard regions and is built on the observation that the reconstruction loss of individual frames in MAM can provide natural signals to judge the difficulty of solving the MAM pre-text task for that frame. To identify these hard regions, we employ a teacher model that first predicts the frame-wise losses and then decides which frames to mask. By learning to create challenging problems, such as identifying harder frames and solving them simultaneously, the model is able to learn more effective representations and thereby acquire a more comprehensive understanding of the speech. Quantitatively, EH-MAM outperforms several state-of-the-art baselines across various low-resource speech recognition and SUPERB benchmarks by 5%-10%. Additionally, we conduct a thorough analysis to show that the regions masked by EH-MAM effectively capture useful context across speech frames.

Related papers

• Semantic Refocused Tuning for Open-Vocabulary Panoptic Segmentation [42.020470627552136]
  Open-vocabulary panoptic segmentation is an emerging task aiming to accurately segment the image into semantically meaningful masks. mask classification is the main performance bottleneck for open-vocab panoptic segmentation. We propose Semantic Refocused Tuning, a novel framework that greatly enhances open-vocab panoptic segmentation.
  arXiv  Detail & Related papers  (2024-09-24T17:50:28Z)
• Joint Semantic Knowledge Distillation and Masked Acoustic Modeling for Full-band Speech Restoration with Improved Intelligibility [15.463932957443973]
  Speech restoration aims at restoring full-band speech with high quality and intelligibility, considering a diverse set of distortions. MaskSR is a recently proposed generative model for this task. We show that, with the same MaskSR model capacity and inference time, the proposed model, MaskSR2, significantly reduces the word error rate, a typical metric for intelligibility.
  arXiv  Detail & Related papers  (2024-09-14T08:09:55Z)
• Towards Latent Masked Image Modeling for Self-Supervised Visual Representation Learning [18.424840375721303]
  Masked Image Modeling (MIM) has emerged as a promising method for deriving visual representations from unlabeled image data by predicting missing pixels from masked portions of images. A promising yet unrealized framework is learning representations through masked reconstruction in latent space, combining the locality of MIM with the high-level targets. This study is among the first to thoroughly analyze and address the challenges of such framework, which we refer to as Latent MIM.
  arXiv  Detail & Related papers  (2024-07-22T17:54:41Z)
• MaskSR: Masked Language Model for Full-band Speech Restoration [7.015213589171985]
  Speech restoration aims at restoring high quality speech in the presence of a diverse set of distortions. We propose MaskSR, a masked language model capable of restoring full-band 44.1 kHz speech jointly considering noise, reverb, clipping, and low bandwidth.
  arXiv  Detail & Related papers  (2024-06-04T08:23:57Z)
• Understanding Masked Autoencoders From a Local Contrastive Perspective [80.57196495601826]
  Masked AutoEncoder (MAE) has revolutionized the field of self-supervised learning with its simple yet effective masking and reconstruction strategies. We introduce a new empirical framework, called Local Contrastive MAE, to analyze both reconstructive and contrastive aspects of MAE.
  arXiv  Detail & Related papers  (2023-10-03T12:08:15Z)
• Hard Patches Mining for Masked Image Modeling [52.46714618641274]
  Masked image modeling (MIM) has attracted much research attention due to its promising potential for learning scalable visual representations. We propose Hard Patches Mining (HPM), a brand-new framework for MIM pre-training.
  arXiv  Detail & Related papers  (2023-04-12T15:38:23Z)
• Improving Masked Autoencoders by Learning Where to Mask [65.89510231743692]
  Masked image modeling is a promising self-supervised learning method for visual data. We present AutoMAE, a framework that uses Gumbel-Softmax to interlink an adversarially-trained mask generator and a mask-guided image modeling process. In our experiments, AutoMAE is shown to provide effective pretraining models on standard self-supervised benchmarks and downstream tasks.
  arXiv  Detail & Related papers  (2023-03-12T05:28:55Z)
• Exploring The Role of Mean Teachers in Self-supervised Masked Auto-Encoders [64.03000385267339]
  Masked image modeling (MIM) has become a popular strategy for self-supervised learning(SSL) of visual representations with Vision Transformers. We present a simple SSL method, the Reconstruction-Consistent Masked Auto-Encoder (RC-MAE) by adding an EMA teacher to MAE. RC-MAE converges faster and requires less memory usage than state-of-the-art self-distillation methods during pre-training.
  arXiv  Detail & Related papers  (2022-10-05T08:08:55Z)
• Masked Vision and Language Modeling for Multi-modal Representation Learning [62.15254888833132]
  We study how to use masked signal modeling in vision and language (V+L) representation learning. We propose to build joint masked vision and language modeling, where the masked signal of one modality is reconstructed with the help from another modality. Our experiments on various V+L tasks show that the proposed method achieves state-of-the-art performances by using a large amount of data.
  arXiv  Detail & Related papers  (2022-08-03T15:11:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.