Local Nonparametric Meta-Learning
- URL: http://arxiv.org/abs/2002.03272v1
- Date: Sun, 9 Feb 2020 03:28:27 GMT
- Title: Local Nonparametric Meta-Learning
- Authors: Wonjoon Goo, Scott Niekum
- Abstract summary: A central goal of meta-learning is to find a learning rule that enables fast adaptation across a set of tasks.
We show that global, fixed-size representations often fail when confronted with certain types of out-of-distribution tasks.
We propose a novel nonparametric meta-learning algorithm that utilizes a meta-trained local learning rule.
- Score: 28.563015766188478
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: A central goal of meta-learning is to find a learning rule that enables fast
adaptation across a set of tasks, by learning the appropriate inductive bias
for that set. Most meta-learning algorithms try to find a \textit{global}
learning rule that encodes this inductive bias. However, a global learning rule
represented by a fixed-size representation is prone to meta-underfitting or
-overfitting since the right representational power for a task set is difficult
to choose a priori. Even when chosen correctly, we show that global, fixed-size
representations often fail when confronted with certain types of
out-of-distribution tasks, even when the same inductive bias is appropriate. To
address these problems, we propose a novel nonparametric meta-learning
algorithm that utilizes a meta-trained local learning rule, building on recent
ideas in attention-based and functional gradient-based meta-learning. In
several meta-regression problems, we show improved meta-generalization results
using our local, nonparametric approach and achieve state-of-the-art results in
the robotics benchmark, Omnipush.
Related papers
- General-Purpose In-Context Learning by Meta-Learning Transformers [45.63069059498147]
We show that Transformers and other black-box models can be meta-trained to act as general-purpose in-context learners.
We characterize transitions between algorithms that generalize, algorithms that memorize, and algorithms that fail to meta-train at all.
We propose practical interventions such as biasing the training distribution that improve the meta-training and meta-generalization of general-purpose in-context learning algorithms.
arXiv Detail & Related papers (2022-12-08T18:30:22Z) - Set-based Meta-Interpolation for Few-Task Meta-Learning [79.4236527774689]
We propose a novel domain-agnostic task augmentation method, Meta-Interpolation, to densify the meta-training task distribution.
We empirically validate the efficacy of Meta-Interpolation on eight datasets spanning across various domains.
arXiv Detail & Related papers (2022-05-20T06:53:03Z) - Learning an Explicit Hyperparameter Prediction Function Conditioned on
Tasks [62.63852372239708]
Meta learning aims to learn the learning methodology for machine learning from observed tasks, so as to generalize to new query tasks.
We interpret such learning methodology as learning an explicit hyper- parameter prediction function shared by all training tasks.
Such setting guarantees that the meta-learned learning methodology is able to flexibly fit diverse query tasks.
arXiv Detail & Related papers (2021-07-06T04:05:08Z) - Meta-Learning with Fewer Tasks through Task Interpolation [67.03769747726666]
Current meta-learning algorithms require a large number of meta-training tasks, which may not be accessible in real-world scenarios.
By meta-learning with task gradient (MLTI), our approach effectively generates additional tasks by randomly sampling a pair of tasks and interpolating the corresponding features and labels.
Empirically, in our experiments on eight datasets from diverse domains, we find that the proposed general MLTI framework is compatible with representative meta-learning algorithms and consistently outperforms other state-of-the-art strategies.
arXiv Detail & Related papers (2021-06-04T20:15:34Z) - Meta-Learning with Neural Tangent Kernels [58.06951624702086]
We propose the first meta-learning paradigm in the Reproducing Kernel Hilbert Space (RKHS) induced by the meta-model's Neural Tangent Kernel (NTK)
Within this paradigm, we introduce two meta-learning algorithms, which no longer need a sub-optimal iterative inner-loop adaptation as in the MAML framework.
We achieve this goal by 1) replacing the adaptation with a fast-adaptive regularizer in the RKHS; and 2) solving the adaptation analytically based on the NTK theory.
arXiv Detail & Related papers (2021-02-07T20:53:23Z) - Variable-Shot Adaptation for Online Meta-Learning [123.47725004094472]
We study the problem of learning new tasks from a small, fixed number of examples, by meta-learning across static data from a set of previous tasks.
We find that meta-learning solves the full task set with fewer overall labels and greater cumulative performance, compared to standard supervised methods.
These results suggest that meta-learning is an important ingredient for building learning systems that continuously learn and improve over a sequence of problems.
arXiv Detail & Related papers (2020-12-14T18:05:24Z) - Improving Generalization in Meta-learning via Task Augmentation [69.83677015207527]
We propose two task augmentation methods, including MetaMix and Channel Shuffle.
Both MetaMix and Channel Shuffle outperform state-of-the-art results by a large margin across many datasets.
arXiv Detail & Related papers (2020-07-26T01:50:42Z) - Guarantees for Tuning the Step Size using a Learning-to-Learn Approach [18.838453594698166]
We give meta-optimization guarantees for the learning-to-learn approach on a simple problem of tuning the step size for quadratic loss.
Although there is a way to design the meta-objective so that the meta-gradient remains bounded, computing the meta-gradient directly using backpropagation leads to numerical issues.
We also characterize when it is necessary to compute the meta-objective on a separate validation set to ensure the performance of the learned.
arXiv Detail & Related papers (2020-06-30T02:59:35Z) - TaskNorm: Rethinking Batch Normalization for Meta-Learning [43.01116858195183]
We evaluate a range of approaches to batch normalization for meta-learning scenarios, and develop a novel approach that we call TaskNorm.
Experiments on fourteen datasets demonstrate that the choice of batch normalization has a dramatic effect on both classification accuracy and training time.
We provide a set of best practices for normalization that will allow fair comparison of meta-learning algorithms.
arXiv Detail & Related papers (2020-03-06T15:43:27Z) - Incremental Meta-Learning via Indirect Discriminant Alignment [118.61152684795178]
We develop a notion of incremental learning during the meta-training phase of meta-learning.
Our approach performs favorably at test time as compared to training a model with the full meta-training set.
arXiv Detail & Related papers (2020-02-11T01:39:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.