Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition
- URL: http://arxiv.org/abs/2106.08922v1
- Date: Wed, 16 Jun 2021 16:24:55 GMT
- Title: Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition
- Authors: Yosuke Higuchi, Niko Moritz, Jonathan Le Roux, Takaaki Hori
- Abstract summary: We present momentum pseudo-labeling (MPL) as a simple yet effective strategy for semi-supervised speech recognition.
MPL consists of a pair of online and offline models that interact and learn from each other, inspired by the mean teacher method.
The experimental results demonstrate that MPL effectively improves over the base model and is scalable to different semi-supervised scenarios.
- Score: 55.362258027878966
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Pseudo-labeling (PL) has been shown to be effective in semi-supervised
automatic speech recognition (ASR), where a base model is self-trained with
pseudo-labels generated from unlabeled data. While PL can be further improved
by iteratively updating pseudo-labels as the model evolves, most of the
previous approaches involve inefficient retraining of the model or intricate
control of the label update. We present momentum pseudo-labeling (MPL), a
simple yet effective strategy for semi-supervised ASR. MPL consists of a pair
of online and offline models that interact and learn from each other, inspired
by the mean teacher method. The online model is trained to predict
pseudo-labels generated on the fly by the offline model. The offline model
maintains a momentum-based moving average of the online model. MPL is performed
in a single training process and the interaction between the two models
effectively helps them reinforce each other to improve the ASR performance. We
apply MPL to an end-to-end ASR model based on the connectionist temporal
classification. The experimental results demonstrate that MPL effectively
improves over the base model and is scalable to different semi-supervised
scenarios with varying amounts of data or domain mismatch.
Related papers
- EMR-Merging: Tuning-Free High-Performance Model Merging [55.03509900949149]
We show that Elect, Mask & Rescale-Merging (EMR-Merging) shows outstanding performance compared to existing merging methods.
EMR-Merging is tuning-free, thus requiring no data availability or any additional training while showing impressive performance.
arXiv Detail & Related papers (2024-05-23T05:25:45Z) - Exploring Beyond Logits: Hierarchical Dynamic Labeling Based on Embeddings for Semi-Supervised Classification [49.09505771145326]
We propose a Hierarchical Dynamic Labeling (HDL) algorithm that does not depend on model predictions and utilizes image embeddings to generate sample labels.
Our approach has the potential to change the paradigm of pseudo-label generation in semi-supervised learning.
arXiv Detail & Related papers (2024-04-26T06:00:27Z) - Semi-Supervised Class-Agnostic Motion Prediction with Pseudo Label
Regeneration and BEVMix [59.55173022987071]
We study the potential of semi-supervised learning for class-agnostic motion prediction.
Our framework adopts a consistency-based self-training paradigm, enabling the model to learn from unlabeled data.
Our method exhibits comparable performance to weakly and some fully supervised methods.
arXiv Detail & Related papers (2023-12-13T09:32:50Z) - AdaMerging: Adaptive Model Merging for Multi-Task Learning [68.75885518081357]
This paper introduces an innovative technique called Adaptive Model Merging (AdaMerging)
It aims to autonomously learn the coefficients for model merging, either in a task-wise or layer-wise manner, without relying on the original training data.
Compared to the current state-of-the-art task arithmetic merging scheme, AdaMerging showcases a remarkable 11% improvement in performance.
arXiv Detail & Related papers (2023-10-04T04:26:33Z) - Balancing Efficiency vs. Effectiveness and Providing Missing Label
Robustness in Multi-Label Stream Classification [3.97048491084787]
We propose a neural network-based approach to high-dimensional multi-label classification.
Our model uses a selective concept drift adaptation mechanism that makes it suitable for a non-stationary environment.
We adapt our model to an environment with missing labels using a simple yet effective imputation strategy.
arXiv Detail & Related papers (2023-10-01T13:23:37Z) - Active Self-Semi-Supervised Learning for Few Labeled Samples [4.713652957384158]
Training deep models with limited annotations poses a significant challenge when applied to diverse practical domains.
We propose a simple yet effective framework, active self-semi-supervised learning (AS3L)
AS3L bootstraps semi-supervised models with prior pseudo-labels (PPL)
We develop active learning and label propagation strategies to obtain accurate PPL.
arXiv Detail & Related papers (2022-03-09T07:45:05Z) - Self-Damaging Contrastive Learning [92.34124578823977]
Unlabeled data in reality is commonly imbalanced and shows a long-tail distribution.
This paper proposes a principled framework called Self-Damaging Contrastive Learning to automatically balance the representation learning without knowing the classes.
Our experiments show that SDCLR significantly improves not only overall accuracies but also balancedness.
arXiv Detail & Related papers (2021-06-06T00:04:49Z) - Adaptive Consistency Regularization for Semi-Supervised Transfer
Learning [31.66745229673066]
We consider semi-supervised learning and transfer learning jointly, leading to a more practical and competitive paradigm.
To better exploit the value of both pre-trained weights and unlabeled target examples, we introduce adaptive consistency regularization.
Our proposed adaptive consistency regularization outperforms state-of-the-art semi-supervised learning techniques such as Pseudo Label, Mean Teacher, and MixMatch.
arXiv Detail & Related papers (2021-03-03T05:46:39Z) - SlimIPL: Language-Model-Free Iterative Pseudo-Labeling [32.39921686482643]
Iterative Pseudo-Labeling (IPL) continuously trains a single model using pseudo-labels iteratively re-generated as the model learns.
We call this approach Language-Model-Free IPL (slimIPL) and give a resultant training setup for low-resource settings with CTC-based models.
arXiv Detail & Related papers (2020-10-22T08:36:33Z) - Variational Model-based Policy Optimization [34.80171122943031]
Model-based reinforcement learning (RL) algorithms allow us to combine model-generated data with those collected from interaction with the real system in order to alleviate the data efficiency problem in RL.
We propose an objective function as a variational lower-bound of a log-likelihood of a log-likelihood to jointly learn and improve model and policy.
Our experiments on a number of continuous control tasks show that despite being more complex, our model-based (E-step) algorithm, called emactoral model-based policy optimization (VMBPO), is more sample-efficient and
arXiv Detail & Related papers (2020-06-09T18:30:15Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.