Kaizen: Continuously improving teacher using Exponential Moving Average for semi-supervised speech recognition

• URL: http://arxiv.org/abs/2106.07759v1
• Date: Mon, 14 Jun 2021 21:15:36 GMT
• Title: Kaizen: Continuously improving teacher using Exponential Moving Average for semi-supervised speech recognition
• Abstract summary: The proposed approach uses a teacher model which is updated as the exponential moving average of the student model parameters. The proposed approach shows more than 10% word error rate (WER) reduction over standard teacher-student training.
• Score: 41.78509185887238
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we introduce the Kaizen framework that uses a continuously improving teacher to generate pseudo-labels for semi-supervised training. The proposed approach uses a teacher model which is updated as the exponential moving average of the student model parameters. This can be seen as a continuous version of the iterative pseudo-labeling approach for semi-supervised training. It is applicable for different training criteria, and in this paper we demonstrate it for frame-level hybrid hidden Markov model - deep neural network (HMM-DNN) models and sequence-level connectionist temporal classification (CTC) based models. The proposed approach shows more than 10% word error rate (WER) reduction over standard teacher-student training and more than 50\% relative WER reduction over 10 hour supervised baseline when using large scale realistic unsupervised public videos in UK English and Italian languages.

Related papers

• Rethinking the Mean Teacher Strategy from the Perspective of Self-paced Learning [5.6818939992896365]
  Semi-supervised medical image segmentation has attracted significant attention due to its potential to reduce manual annotation costs.<n>In this work, we reinterpret the MT strategy on supervised data as a form of self-paced learning, regulated by the output agreement between the temporally lagged teacher model and the ground truth labels.
  arXiv  Detail & Related papers  (2025-05-16T09:14:06Z)
• PMT: Progressive Mean Teacher via Exploring Temporal Consistency for Semi-Supervised Medical Image Segmentation [51.509573838103854]
  We propose a semi-supervised learning framework, termed Progressive Mean Teachers (PMT), for medical image segmentation. Our PMT generates high-fidelity pseudo labels by learning robust and diverse features in the training process. Experimental results on two datasets with different modalities, i.e., CT and MRI, demonstrate that our method outperforms the state-of-the-art medical image segmentation approaches.
  arXiv  Detail & Related papers  (2024-09-08T15:02:25Z)
• Learning from Future: A Novel Self-Training Framework for Semantic Segmentation [33.66516999361252]
  Self-training has shown great potential in semi-supervised learning. We propose a novel self-training strategy, which allows the model to learn from the future. We experimentally demonstrate the effectiveness and superiority of our approach under a wide range of settings.
  arXiv  Detail & Related papers  (2022-09-15T01:39:46Z)
• Adapting the Mean Teacher for keypoint-based lung registration under geometric domain shifts [75.51482952586773]
  deep neural networks generally require plenty of labeled training data and are vulnerable to domain shifts between training and test data. We present a novel approach to geometric domain adaptation for image registration, adapting a model from a labeled source to an unlabeled target domain. Our method consistently improves on the baseline model by 50%/47% while even matching the accuracy of models trained on target data.
  arXiv  Detail & Related papers  (2022-07-01T12:16:42Z)
• Language Models in the Loop: Incorporating Prompting into Weak Supervision [11.10422546502386]
  We propose a new strategy for applying large pre-trained language models to novel tasks when labeled training data is limited. Instead of applying the model in a typical zero-shot or few-shot fashion, we treat the model as the basis for labeling functions in a weak supervision framework.
  arXiv  Detail & Related papers  (2022-05-04T20:42:40Z)
• Momentum Pseudo-Labeling for Semi-Supervised Speech Recognition [55.362258027878966]
  We present momentum pseudo-labeling (MPL) as a simple yet effective strategy for semi-supervised speech recognition. MPL consists of a pair of online and offline models that interact and learn from each other, inspired by the mean teacher method. The experimental results demonstrate that MPL effectively improves over the base model and is scalable to different semi-supervised scenarios.
  arXiv  Detail & Related papers  (2021-06-16T16:24:55Z)
• Unsupervised Class-Incremental Learning Through Confusion [0.4604003661048266]
  We introduce a novelty detection method that leverages network confusion caused by training incoming data as a new class. We found that incorporating a class-imbalance during this detection method substantially enhances performance.
  arXiv  Detail & Related papers  (2021-04-09T15:58:43Z)
• Exponential Moving Average Normalization for Self-supervised and Semi-supervised Learning [90.69256802784689]
  We present a plug-in replacement for batch normalization (BN) called exponential moving average normalization (EMAN) Unlike the standard BN, where the statistics are computed within each batch, EMAN, used in the teacher, updates its statistics by exponential moving average from the BN statistics of the student. EMAN improves strong baselines for self-supervised learning by 4-6/1-2 points and semi-supervised learning by about 7/2 points, when 1%/10% supervised labels are available on ImageNet.
  arXiv  Detail & Related papers  (2021-01-21T07:45:37Z)
• Self-Supervised Contrastive Learning for Unsupervised Phoneme Segmentation [37.054709598792165]
  The model is a convolutional neural network that operates directly on the raw waveform. It is optimized to identify spectral changes in the signal using the Noise-Contrastive Estimation principle. At test time, a peak detection algorithm is applied over the model outputs to produce the final boundaries.
  arXiv  Detail & Related papers  (2020-07-27T12:10:21Z)
• Learning to Reweight with Deep Interactions [104.68509759134878]
  We propose an improved data reweighting algorithm, in which the student model provides its internal states to the teacher model. Experiments on image classification with clean/noisy labels and neural machine translation empirically demonstrate that our algorithm makes significant improvement over previous methods.
  arXiv  Detail & Related papers  (2020-07-09T09:06:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.