Improved Meta Learning for Low Resource Speech Recognition

• URL: http://arxiv.org/abs/2205.06182v1
• Date: Wed, 11 May 2022 15:50:47 GMT
• Title: Improved Meta Learning for Low Resource Speech Recognition
• Authors: Satwinder Singh, Ruili Wang, Feng Hou
• Abstract summary: We propose a new meta learning based framework for low resource speech recognition that improves the previous model meta learning (MAML) approach. Our proposed system outperforms MAML based low resource ASR system on various languages in terms of character error rates and stable training behavior.
• Score: 15.612232220719653
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose a new meta learning based framework for low resource speech recognition that improves the previous model agnostic meta learning (MAML) approach. The MAML is a simple yet powerful meta learning approach. However, the MAML presents some core deficiencies such as training instabilities and slower convergence speed. To address these issues, we adopt multi-step loss (MSL). The MSL aims to calculate losses at every step of the inner loop of MAML and then combines them with a weighted importance vector. The importance vector ensures that the loss at the last step has more importance than the previous steps. Our empirical evaluation shows that MSL significantly improves the stability of the training procedure and it thus also improves the accuracy of the overall system. Our proposed system outperforms MAML based low resource ASR system on various languages in terms of character error rates and stable training behavior.

Related papers

• Understanding the Role of LLMs in Multimodal Evaluation Benchmarks [77.59035801244278]
  This paper investigates the role of the Large Language Model (LLM) backbone in Multimodal Large Language Models (MLLMs) evaluation. Our study encompasses four diverse MLLM benchmarks and eight state-of-the-art MLLMs. Key findings reveal that some benchmarks allow high performance even without visual inputs and up to 50% of error rates can be attributed to insufficient world knowledge in the LLM backbone.
  arXiv  Detail & Related papers  (2024-10-16T07:49:13Z)
• CoMMIT: Coordinated Instruction Tuning for Multimodal Large Language Models [68.64605538559312]
  In this paper, we analyze the MLLM instruction tuning from both theoretical and empirical perspectives. Inspired by our findings, we propose a measurement to quantitatively evaluate the learning balance. In addition, we introduce an auxiliary loss regularization method to promote updating of the generation distribution of MLLMs.
  arXiv  Detail & Related papers  (2024-07-29T23:18:55Z)
• Provable Generalization of Overparameterized Meta-learning Trained with SGD [62.892930625034374]
  We study the generalization of a widely used meta-learning approach, Model-Agnostic Meta-Learning (MAML) We provide both upper and lower bounds for the excess risk of MAML, which captures how SGD dynamics affect these generalization bounds. Our theoretical findings are further validated by experiments.
  arXiv  Detail & Related papers  (2022-06-18T07:22:57Z)
• Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learning [71.26635165491105]
  We develop a sharpness-aware MAML approach that we term Sharp-MAML. We empirically demonstrate that Sharp-MAML and its computation-efficient variant can outperform popular existing MAML baselines. This is the first empirical and theoretical study on sharpness-aware minimization in the context of bilevel learning.
  arXiv  Detail & Related papers  (2022-06-08T16:20:11Z)
• MAML is a Noisy Contrastive Learner [72.04430033118426]
  Model-agnostic meta-learning (MAML) is one of the most popular and widely-adopted meta-learning algorithms nowadays. We provide a new perspective to the working mechanism of MAML and discover that: MAML is analogous to a meta-learner using a supervised contrastive objective function. We propose a simple but effective technique, zeroing trick, to alleviate such interference.
  arXiv  Detail & Related papers  (2021-06-29T12:52:26Z)
• Repurposing Pretrained Models for Robust Out-of-domain Few-Shot Learning [23.135033752967598]
  We consider the novel problem of repurposing pretrained MAML checkpoints to solve new few-shot classification tasks. Because of the potential distribution mismatch, the original MAML steps may no longer be optimal. We propose an alternative metatesting procedure and combine adversarial training and uncertainty-based stepsize adaptation.
  arXiv  Detail & Related papers  (2021-03-16T12:53:09Z)
• On Fast Adversarial Robustness Adaptation in Model-Agnostic Meta-Learning [100.14809391594109]
  Model-agnostic meta-learning (MAML) has emerged as one of the most successful meta-learning techniques in few-shot learning. Despite the generalization power of the meta-model, it remains elusive that how adversarial robustness can be maintained by MAML in few-shot learning. We propose a general but easily-optimized robustness-regularized meta-learning framework, which allows the use of unlabeled data augmentation, fast adversarial attack generation, and computationally-light fine-tuning.
  arXiv  Detail & Related papers  (2021-02-20T22:03:04Z)
• B-SMALL: A Bayesian Neural Network approach to Sparse Model-Agnostic Meta-Learning [2.9189409618561966]
  We propose a Bayesian neural network based MAML algorithm, which we refer to as the B-SMALL algorithm. We demonstrate the performance of B-MAML using classification and regression tasks, and highlight that training a sparsifying BNN using MAML indeed improves the parameter footprint of the model.
  arXiv  Detail & Related papers  (2021-01-01T09:19:48Z)
• La-MAML: Look-ahead Meta Learning for Continual Learning [14.405620521842621]
  We propose Look-ahead MAML (La-MAML), a fast optimisation-based meta-learning algorithm for online-continual learning, aided by a small episodic memory. La-MAML achieves performance superior to other replay-based, prior-based and meta-learning based approaches for continual learning on real-world visual classification benchmarks.
  arXiv  Detail & Related papers  (2020-07-27T23:07:01Z)
• Weighted Meta-Learning [21.522768804834616]
  Many popular meta-learning algorithms, such as model-agnostic meta-learning (MAML), only assume access to the target samples for fine-tuning. In this work, we provide a general framework for meta-learning based on weighting the loss of different source tasks. We develop a learning algorithm based on minimizing the error bound with respect to an empirical IPM, including a weighted MAML algorithm.
  arXiv  Detail & Related papers  (2020-03-20T19:00:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.