Is Bayesian Model-Agnostic Meta Learning Better than Model-Agnostic Meta Learning, Provably?

• URL: http://arxiv.org/abs/2203.03059v1
• Date: Sun, 6 Mar 2022 21:38:18 GMT
• Title: Is Bayesian Model-Agnostic Meta Learning Better than Model-Agnostic Meta Learning, Provably?
• Authors: Lisha Chen, Tianyi Chen
• Abstract summary: We compare the meta test risks of model agnostic meta learning (MAML) and Bayesian MAML. Under both the distribution agnostic and linear centroid cases, we have established that Bayesian MAML indeed has provably lower meta test risks than MAML.
• Score: 25.00480072097939
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Meta learning aims at learning a model that can quickly adapt to unseen tasks. Widely used meta learning methods include model agnostic meta learning (MAML), implicit MAML, Bayesian MAML. Thanks to its ability of modeling uncertainty, Bayesian MAML often has advantageous empirical performance. However, the theoretical understanding of Bayesian MAML is still limited, especially on questions such as if and when Bayesian MAML has provably better performance than MAML. In this paper, we aim to provide theoretical justifications for Bayesian MAML's advantageous performance by comparing the meta test risks of MAML and Bayesian MAML. In the meta linear regression, under both the distribution agnostic and linear centroid cases, we have established that Bayesian MAML indeed has provably lower meta test risks than MAML. We verify our theoretical results through experiments.

Related papers

• Provable Generalization of Overparameterized Meta-learning Trained with SGD [62.892930625034374]
  We study the generalization of a widely used meta-learning approach, Model-Agnostic Meta-Learning (MAML) We provide both upper and lower bounds for the excess risk of MAML, which captures how SGD dynamics affect these generalization bounds. Our theoretical findings are further validated by experiments.
  arXiv  Detail & Related papers  (2022-06-18T07:22:57Z)
• Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learning [71.26635165491105]
  We develop a sharpness-aware MAML approach that we term Sharp-MAML. We empirically demonstrate that Sharp-MAML and its computation-efficient variant can outperform popular existing MAML baselines. This is the first empirical and theoretical study on sharpness-aware minimization in the context of bilevel learning.
  arXiv  Detail & Related papers  (2022-06-08T16:20:11Z)
• Sign-MAML: Efficient Model-Agnostic Meta-Learning by SignSGD [16.417263188143313]
  We propose a new computationally-efficient first-order algorithm for Model-Agnostic Meta-Learning (MAML) We show that MAML, through the lens of signSGD-oriented BLO, naturally yields an alternating optimization scheme that just requires first-order gradients of a learned meta-model. In practice, we show that Sign-MAML outperforms FO-MAML in various few-shot image classification tasks.
  arXiv  Detail & Related papers  (2021-09-15T18:01:55Z)
• MAML is a Noisy Contrastive Learner [72.04430033118426]
  Model-agnostic meta-learning (MAML) is one of the most popular and widely-adopted meta-learning algorithms nowadays. We provide a new perspective to the working mechanism of MAML and discover that: MAML is analogous to a meta-learner using a supervised contrastive objective function. We propose a simple but effective technique, zeroing trick, to alleviate such interference.
  arXiv  Detail & Related papers  (2021-06-29T12:52:26Z)
• Memory-Based Optimization Methods for Model-Agnostic Meta-Learning and Personalized Federated Learning [56.17603785248675]
  Model-agnostic meta-learning (MAML) has become a popular research area. Existing MAML algorithms rely on the episode' idea by sampling a few tasks and data points to update the meta-model at each iteration. This paper proposes memory-based algorithms for MAML that converge with vanishing error.
  arXiv  Detail & Related papers  (2021-06-09T08:47:58Z)
• On Fast Adversarial Robustness Adaptation in Model-Agnostic Meta-Learning [100.14809391594109]
  Model-agnostic meta-learning (MAML) has emerged as one of the most successful meta-learning techniques in few-shot learning. Despite the generalization power of the meta-model, it remains elusive that how adversarial robustness can be maintained by MAML in few-shot learning. We propose a general but easily-optimized robustness-regularized meta-learning framework, which allows the use of unlabeled data augmentation, fast adversarial attack generation, and computationally-light fine-tuning.
  arXiv  Detail & Related papers  (2021-02-20T22:03:04Z)
• Meta Learning in the Continuous Time Limit [36.23467808322093]
  We establish the ordinary differential equation (ODE) that underlies the training dynamics of Model-A Meta-Learning (MAML) We propose a new BI-MAML training algorithm that significantly reduces the computational burden associated with existing MAML training methods.
  arXiv  Detail & Related papers  (2020-06-19T01:47:31Z)
• When does MAML Work the Best? An Empirical Study on Model-Agnostic Meta-Learning in NLP Applications [26.458825286934857]
  Many impacting factors, including data quantity, similarity among tasks, and the balance between general language model and task-specific adaptation, can affect the performance of MAML in NLP. In this paper, we conduct an empirical study to investigate these impacting factors and conclude when MAML works the best based on the experimental results.
  arXiv  Detail & Related papers  (2020-05-24T09:29:36Z)
• Theoretical Convergence of Multi-Step Model-Agnostic Meta-Learning [63.64636047748605]
  We develop a new theoretical framework to provide convergence guarantee for the general multi-step MAML algorithm. In particular, our results suggest that an inner-stage step needs to be chosen inversely proportional to $N$ of inner-stage steps in order for $N$ MAML to have guaranteed convergence.
  arXiv  Detail & Related papers  (2020-02-18T19:17:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.