HyperMAML: Few-Shot Adaptation of Deep Models with Hypernetworks

• URL: http://arxiv.org/abs/2205.15745v3
• Date: Mon, 8 Jul 2024 14:21:59 GMT
• Title: HyperMAML: Few-Shot Adaptation of Deep Models with Hypernetworks
• Authors: M. Przewięźlikowski, P. Przybysz, J. Tabor, M. Zięba, P. Spurek,
• Abstract summary: Few-Shot learning aims to train models which can easily adapt to previously unseen tasks. Model-Agnostic Meta-Learning (MAML) is one of the most popular Few-Shot learning approaches. In this paper, we propose HyperMAML, where the training of the update procedure is also part of the model.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The aim of Few-Shot learning methods is to train models which can easily adapt to previously unseen tasks, based on small amounts of data. One of the most popular and elegant Few-Shot learning approaches is Model-Agnostic Meta-Learning (MAML). The main idea behind this method is to learn the general weights of the meta-model, which are further adapted to specific problems in a small number of gradient steps. However, the model's main limitation lies in the fact that the update procedure is realized by gradient-based optimisation. In consequence, MAML cannot always modify weights to the essential level in one or even a few gradient iterations. On the other hand, using many gradient steps results in a complex and time-consuming optimization procedure, which is hard to train in practice, and may lead to overfitting. In this paper, we propose HyperMAML, a novel generalization of MAML, where the training of the update procedure is also part of the model. Namely, in HyperMAML, instead of updating the weights with gradient descent, we use for this purpose a trainable Hypernetwork. Consequently, in this framework, the model can generate significant updates whose range is not limited to a fixed number of gradient steps. Experiments show that HyperMAML consistently outperforms MAML and performs comparably to other state-of-the-art techniques in a number of standard Few-Shot learning benchmarks.

Related papers

• Classifier-guided Gradient Modulation for Enhanced Multimodal Learning [50.7008456698935]
  Gradient-Guided Modulation (CGGM) is a novel method to balance multimodal learning with gradients. We conduct extensive experiments on four multimodal datasets: UPMC-Food 101, CMU-MOSI, IEMOCAP and BraTS. CGGM outperforms all the baselines and other state-of-the-art methods consistently.
  arXiv  Detail & Related papers  (2024-11-03T02:38:43Z)
• Hypernetwork approach to Bayesian MAML [13.012692001087617]
  We propose a novel framework for Bayesian MAML called BayesianHMAML. It learns the universal weights point-wise, but a probabilistic structure is added when adapted for specific tasks.
  arXiv  Detail & Related papers  (2022-10-06T10:14:41Z)
• Model-Agnostic Multitask Fine-tuning for Few-shot Vision-Language Transfer Learning [59.38343286807997]
  We propose Model-Agnostic Multitask Fine-tuning (MAMF) for vision-language models on unseen tasks. Compared with model-agnostic meta-learning (MAML), MAMF discards the bi-level optimization and uses only first-order gradients. We show that MAMF consistently outperforms the classical fine-tuning method for few-shot transfer learning on five benchmark datasets.
  arXiv  Detail & Related papers  (2022-03-09T17:26:53Z)
• One Step at a Time: Pros and Cons of Multi-Step Meta-Gradient Reinforcement Learning [61.662504399411695]
  We introduce a novel method mixing multiple inner steps that enjoys a more accurate and robust meta-gradient signal. When applied to the Snake game, the mixing meta-gradient algorithm can cut the variance by a factor of 3 while achieving similar or higher performance.
  arXiv  Detail & Related papers  (2021-10-30T08:36:52Z)
• Contextual Gradient Scaling for Few-Shot Learning [24.19934081878197]
  We propose contextual gradient scaling (CxGrad) for model-agnostic meta-learning (MAML) CxGrad scales gradient norms of the backbone to facilitate learning task-specific knowledge in the inner-loop. Experimental results show that CxGrad effectively encourages the backbone to learn task-specific knowledge in the inner-loop.
  arXiv  Detail & Related papers  (2021-10-20T03:05:58Z)
• Repurposing Pretrained Models for Robust Out-of-domain Few-Shot Learning [23.135033752967598]
  We consider the novel problem of repurposing pretrained MAML checkpoints to solve new few-shot classification tasks. Because of the potential distribution mismatch, the original MAML steps may no longer be optimal. We propose an alternative metatesting procedure and combine adversarial training and uncertainty-based stepsize adaptation.
  arXiv  Detail & Related papers  (2021-03-16T12:53:09Z)
• B-SMALL: A Bayesian Neural Network approach to Sparse Model-Agnostic Meta-Learning [2.9189409618561966]
  We propose a Bayesian neural network based MAML algorithm, which we refer to as the B-SMALL algorithm. We demonstrate the performance of B-MAML using classification and regression tasks, and highlight that training a sparsifying BNN using MAML indeed improves the parameter footprint of the model.
  arXiv  Detail & Related papers  (2021-01-01T09:19:48Z)
• Meta-Learning with Adaptive Hyperparameters [55.182841228303225]
  We focus on a complementary factor in MAML framework, inner-loop optimization (or fast adaptation) We propose a new weight update rule that greatly enhances the fast adaptation process.
  arXiv  Detail & Related papers  (2020-10-31T08:05:34Z)
• La-MAML: Look-ahead Meta Learning for Continual Learning [14.405620521842621]
  We propose Look-ahead MAML (La-MAML), a fast optimisation-based meta-learning algorithm for online-continual learning, aided by a small episodic memory. La-MAML achieves performance superior to other replay-based, prior-based and meta-learning based approaches for continual learning on real-world visual classification benchmarks.
  arXiv  Detail & Related papers  (2020-07-27T23:07:01Z)
• Transfer Learning without Knowing: Reprogramming Black-box Machine Learning Models with Scarce Data and Limited Resources [78.72922528736011]
  We propose a novel approach, black-box adversarial reprogramming (BAR), that repurposes a well-trained black-box machine learning model. Using zeroth order optimization and multi-label mapping techniques, BAR can reprogram a black-box ML model solely based on its input-output responses. BAR outperforms state-of-the-art methods and yields comparable performance to the vanilla adversarial reprogramming method.
  arXiv  Detail & Related papers  (2020-07-17T01:52:34Z)
• Regularizing Meta-Learning via Gradient Dropout [102.29924160341572]
  meta-learning models are prone to overfitting when there are no sufficient training tasks for the meta-learners to generalize. We introduce a simple yet effective method to alleviate the risk of overfitting for gradient-based meta-learning.
  arXiv  Detail & Related papers  (2020-04-13T10:47:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.