Accelerating Gradient-based Meta Learner

• URL: http://arxiv.org/abs/2110.14459v1
• Date: Wed, 27 Oct 2021 14:27:36 GMT
• Title: Accelerating Gradient-based Meta Learner
• Authors: Varad Pimpalkhute, Amey Pandit, Mayank Mishra, Rekha Singhal
• Abstract summary: We propose various acceleration techniques to speed up meta learning algorithms such as MAML (Model Agnostic Meta Learning) We introduce a novel method of training tasks in clusters, which not only accelerates the meta learning process but also improves model accuracy performance.
• Score: 2.1349209400003932
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Meta Learning has been in focus in recent years due to the meta-learner model's ability to adapt well and generalize to new tasks, thus, reducing both the time and data requirements for learning. However, a major drawback of meta learner is that, to reach to a state from where learning new tasks becomes feasible with less data, it requires a large number of iterations and a lot of time. We address this issue by proposing various acceleration techniques to speed up meta learning algorithms such as MAML (Model Agnostic Meta Learning). We present 3.73X acceleration on a well known RNN optimizer based meta learner proposed in literature [11]. We introduce a novel method of training tasks in clusters, which not only accelerates the meta learning process but also improves model accuracy performance. Keywords: Meta learning, RNN optimizer, AGI, Performance optimization

Related papers

• Fast Adaptation with Kernel and Gradient based Meta Leaning [4.763682200721131]
  We propose two algorithms to improve both the inner and outer loops of Model A Meta Learning (MAML) Our first algorithm redefines the optimization problem in the function space to update the model using closed-form solutions. In the outer loop, the second algorithm adjusts the learning of the meta-learner by assigning weights to the losses from each task of the inner loop.
  arXiv  Detail & Related papers  (2024-11-01T07:05:03Z)
• FREE: Faster and Better Data-Free Meta-Learning [77.90126669914324]
  Data-Free Meta-Learning (DFML) aims to extract knowledge from a collection of pre-trained models without requiring the original data. We introduce the Faster and Better Data-Free Meta-Learning framework, which contains: (i) a meta-generator for rapidly recovering training tasks from pre-trained models; and (ii) a meta-learner for generalizing to new unseen tasks.
  arXiv  Detail & Related papers  (2024-05-02T03:43:19Z)
• Learning to Learn with Indispensable Connections [6.040904021861969]
  We propose a novel meta-learning method called Meta-LTH that includes indispensible (necessary) connections. Our method improves the classification accuracy by approximately 2% (20-way 1-shot task setting) for omniglot dataset.
  arXiv  Detail & Related papers  (2023-04-06T04:53:13Z)
• Meta-Learning with Self-Improving Momentum Target [72.98879709228981]
  We propose Self-improving Momentum Target (SiMT) to improve the performance of a meta-learner. SiMT generates the target model by adapting from the temporal ensemble of the meta-learner. We show that SiMT brings a significant performance gain when combined with a wide range of meta-learning methods.
  arXiv  Detail & Related papers  (2022-10-11T06:45:15Z)
• Bootstrapped Meta-Learning [48.017607959109924]
  We propose an algorithm that tackles a challenging meta-optimisation problem by letting the meta-learner teach itself. The algorithm first bootstraps a target from the meta-learner, then optimises the meta-learner by minimising the distance to that target under a chosen (pseudo-)metric. We achieve a new state-of-the art for model-free agents on the Atari ALE benchmark, improve upon MAML in few-shot learning, and demonstrate how our approach opens up new possibilities.
  arXiv  Detail & Related papers  (2021-09-09T18:29:05Z)
• Faster Meta Update Strategy for Noise-Robust Deep Learning [62.08964100618873]
  We introduce a novel Faster Meta Update Strategy (FaMUS) to replace the most expensive step in the meta gradient with a faster layer-wise approximation. We show our method is able to save two-thirds of the training time while still maintaining the comparable or achieving even better generalization performance.
  arXiv  Detail & Related papers  (2021-04-30T16:19:07Z)
• Fast Few-Shot Classification by Few-Iteration Meta-Learning [173.32497326674775]
  We introduce a fast optimization-based meta-learning method for few-shot classification. Our strategy enables important aspects of the base learner objective to be learned during meta-training. We perform a comprehensive experimental analysis, demonstrating the speed and effectiveness of our approach.
  arXiv  Detail & Related papers  (2020-10-01T15:59:31Z)
• La-MAML: Look-ahead Meta Learning for Continual Learning [14.405620521842621]
  We propose Look-ahead MAML (La-MAML), a fast optimisation-based meta-learning algorithm for online-continual learning, aided by a small episodic memory. La-MAML achieves performance superior to other replay-based, prior-based and meta-learning based approaches for continual learning on real-world visual classification benchmarks.
  arXiv  Detail & Related papers  (2020-07-27T23:07:01Z)
• Unraveling Meta-Learning: Understanding Feature Representations for Few-Shot Tasks [55.66438591090072]
  We develop a better understanding of the underlying mechanics of meta-learning and the difference between models trained using meta-learning and models trained classically. We develop a regularizer which boosts the performance of standard training routines for few-shot classification.
  arXiv  Detail & Related papers  (2020-02-17T03:18:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.