Lessons from Chasing Few-Shot Learning Benchmarks: Rethinking the Evaluation of Meta-Learning Methods

• URL: http://arxiv.org/abs/2102.11503v1
• Date: Tue, 23 Feb 2021 05:34:30 GMT
• Title: Lessons from Chasing Few-Shot Learning Benchmarks: Rethinking the Evaluation of Meta-Learning Methods
• Authors: Amrith Setlur, Oscar Li, Virginia Smith
• Abstract summary: We introduce a simple baseline for meta-learning, FIX-ML. We explore two possible goals of meta-learning: to develop methods that generalize (i) to the same task distribution that generates the training set (in-distribution), or (ii) to new, unseen task distributions (out-of-distribution) Our results highlight that in order to reason about progress in this space, it is necessary to provide a clearer description of the goals of meta-learning, and to develop more appropriate evaluation strategies.
• Score: 9.821362920940631
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work we introduce a simple baseline for meta-learning. Our unconventional method, FIX-ML, reduces task diversity by keeping support sets fixed across tasks, and consistently improves the performance of meta-learning methods on popular few-shot learning benchmarks. However, in exploring the reason for this counter-intuitive phenomenon, we unearth a series of questions and concerns about meta-learning evaluation practices. We explore two possible goals of meta-learning: to develop methods that generalize (i) to the same task distribution that generates the training set (in-distribution), or (ii) to new, unseen task distributions (out-of-distribution). Through careful analyses, we show that for each of these two goals, current few-shot learning benchmarks have potential pitfalls in 1) performing model selection and hyperparameter tuning for a given meta-learning method and 2) comparing the performance of different meta-learning methods. Our results highlight that in order to reason about progress in this space, it is necessary to provide a clearer description of the goals of meta-learning, and to develop more appropriate corresponding evaluation strategies.

Related papers

• Meta-Learning with Self-Improving Momentum Target [72.98879709228981]
  We propose Self-improving Momentum Target (SiMT) to improve the performance of a meta-learner. SiMT generates the target model by adapting from the temporal ensemble of the meta-learner. We show that SiMT brings a significant performance gain when combined with a wide range of meta-learning methods.
  arXiv  Detail & Related papers  (2022-10-11T06:45:15Z)
• Diverse Distributions of Self-Supervised Tasks for Meta-Learning in NLP [39.457091182683406]
  We aim to provide task distributions for meta-learning by considering self-supervised tasks automatically proposed from unlabeled text. Our analysis shows that all these factors meaningfully alter the task distribution, some inducing significant improvements in downstream few-shot accuracy of the meta-learned models.
  arXiv  Detail & Related papers  (2021-11-02T01:50:09Z)
• MetaICL: Learning to Learn In Context [87.23056864536613]
  We introduce MetaICL, a new meta-training framework for few-shot learning where a pretrained language model is tuned to do in-context learn-ing on a large set of training tasks. We show that MetaICL approaches (and sometimes beats) the performance of models fully finetuned on the target task training data, and outperforms much bigger models with nearly 8x parameters.
  arXiv  Detail & Related papers  (2021-10-29T17:42:08Z)
• Similarity of Classification Tasks [46.78404360210806]
  We propose a generative approach to analyse task similarity to optimise and better understand the performance of meta-learning. We show that the proposed method can provide an insightful evaluation for meta-learning algorithms on two few-shot classification benchmarks.
  arXiv  Detail & Related papers  (2021-01-27T04:37:34Z)
• Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning [79.25478727351604]
  We explore a simple process: meta-learning over a whole-classification pre-trained model on its evaluation metric. We observe this simple method achieves competitive performance to state-of-the-art methods on standard benchmarks.
  arXiv  Detail & Related papers  (2020-03-09T20:06:36Z)
• Structured Prediction for Conditional Meta-Learning [44.30857707980074]
  We propose a new perspective on conditional meta-learning via structured prediction. We derive task-adaptive structured meta-learning (TASML), a principled framework that yields task-specific objective functions. Empirically, we show that TASML improves the performance of existing meta-learning models, and outperforms the state-of-the-art on benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-20T15:24:15Z)
• Incremental Meta-Learning via Indirect Discriminant Alignment [118.61152684795178]
  We develop a notion of incremental learning during the meta-training phase of meta-learning. Our approach performs favorably at test time as compared to training a model with the full meta-training set.
  arXiv  Detail & Related papers  (2020-02-11T01:39:12Z)
• Revisiting Meta-Learning as Supervised Learning [69.2067288158133]
  We aim to provide a principled, unifying framework by revisiting and strengthening the connection between meta-learning and traditional supervised learning. By treating pairs of task-specific data sets and target models as (feature, label) samples, we can reduce many meta-learning algorithms to instances of supervised learning. This view not only unifies meta-learning into an intuitive and practical framework but also allows us to transfer insights from supervised learning directly to improve meta-learning.
  arXiv  Detail & Related papers  (2020-02-03T06:13:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.