The Curse of Zero Task Diversity: On the Failure of Transfer Learning to Outperform MAML and their Empirical Equivalence

• URL: http://arxiv.org/abs/2112.13121v1
• Date: Fri, 24 Dec 2021 18:42:58 GMT
• Title: The Curse of Zero Task Diversity: On the Failure of Transfer Learning to Outperform MAML and their Empirical Equivalence
• Authors: Brando Miranda, Yu-Xiong Wang and Sanmi Koyejo
• Abstract summary: A transfer learning solution might be all we needed to solve many few-shot learning benchmarks. We name this metric the diversity coefficient of a few-shot learning benchmark. We show that when making a fair comparison between MAML learned solutions to transfer learning, both have identical meta-test accuracy.
• Score: 19.556093984142418
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: It has been recently observed that a transfer learning solution might be all we needed to solve many few-shot learning benchmarks. This raises important questions about when and how meta-learning algorithms should be deployed. In this paper, we make a first step in clarifying these questions by first formulating a computable metric for a few-shot learning benchmark that we hypothesize is predictive of whether meta-learning solutions will succeed or not. We name this metric the diversity coefficient of a few-shot learning benchmark. Using the diversity coefficient, we show that the MiniImagenet benchmark has zero diversity - according to twenty-four different ways to compute the diversity. We proceed to show that when making a fair comparison between MAML learned solutions to transfer learning, both have identical meta-test accuracy. This suggests that transfer learning fails to outperform MAML - contrary to what previous work suggests. Together, these two facts provide the first test of whether diversity correlates with meta-learning success and therefore show that a diversity coefficient of zero correlates with a high similarity between transfer learning and MAML learned solutions - especially at meta-test time. We therefore conjecture meta-learned solutions have the same meta-test performance as transfer learning when the diversity coefficient is zero.

Related papers

• CLOSER: Towards Better Representation Learning for Few-Shot Class-Incremental Learning [52.63674911541416]
  Few-shot class-incremental learning (FSCIL) faces several challenges, such as overfitting and forgetting. Our primary focus is representation learning on base classes to tackle the unique challenge of FSCIL. We find that trying to secure the spread of features within a more confined feature space enables the learned representation to strike a better balance between transferability and discriminability.
  arXiv  Detail & Related papers  (2024-10-08T02:23:16Z)
• Learning to Learn with Indispensable Connections [6.040904021861969]
  We propose a novel meta-learning method called Meta-LTH that includes indispensible (necessary) connections. Our method improves the classification accuracy by approximately 2% (20-way 1-shot task setting) for omniglot dataset.
  arXiv  Detail & Related papers  (2023-04-06T04:53:13Z)
• The Curse of Low Task Diversity: On the Failure of Transfer Learning to Outperform MAML and Their Empirical Equivalence [20.965759895300327]
  We propose a novel metric -- the diversity coefficient -- to measure the diversity of tasks in a few-shot learning benchmark. Using the diversity coefficient, we show that the popular MiniImageNet and CIFAR-FS few-shot learning benchmarks have low diversity.
  arXiv  Detail & Related papers  (2022-08-02T15:49:11Z)
• The Effect of Diversity in Meta-Learning [79.56118674435844]
  Few-shot learning aims to learn representations that can tackle novel tasks given a small number of examples. Recent studies show that task distribution plays a vital role in the model's performance. We study different task distributions on a myriad of models and datasets to evaluate the effect of task diversity on meta-learning algorithms.
  arXiv  Detail & Related papers  (2022-01-27T19:39:07Z)
• Does MAML Only Work via Feature Re-use? A Data Centric Perspective [19.556093984142418]
  We provide empirical results that shed some light on how meta-learned MAML representations function. We show that it is possible to define a family of synthetic benchmarks that result in a low degree of feature re-use. We conjecture the core challenge of re-thinking meta-learning is in the design of few-shot learning data sets and benchmarks.
  arXiv  Detail & Related papers  (2021-12-24T20:18:38Z)
• Meta-learning Amidst Heterogeneity and Ambiguity [11.061517140668961]
  We devise a novel meta-learning framework, called Meta-learning Amidst Heterogeneity and Ambiguity (MAHA) By extensively conducting several experiments in regression and classification, we demonstrate the validity of our model.
  arXiv  Detail & Related papers  (2021-07-05T18:54:31Z)
• MAML is a Noisy Contrastive Learner [72.04430033118426]
  Model-agnostic meta-learning (MAML) is one of the most popular and widely-adopted meta-learning algorithms nowadays. We provide a new perspective to the working mechanism of MAML and discover that: MAML is analogous to a meta-learner using a supervised contrastive objective function. We propose a simple but effective technique, zeroing trick, to alleviate such interference.
  arXiv  Detail & Related papers  (2021-06-29T12:52:26Z)
• Comparing Transfer and Meta Learning Approaches on a Unified Few-Shot Classification Benchmark [44.530605715850506]
  Cross-family study of the best transfer and meta learners on a large-scale meta-learning benchmark and a transfer learning benchmark. We find that, on average, large-scale transfer methods (Big Transfer, BiT) outperform competing approaches on MD, even when trained only on ImageNet. We reveal a number of discrepancies in evaluation norms and study some of these in light of the performance gap.
  arXiv  Detail & Related papers  (2021-04-06T16:17:51Z)
• A Taxonomy of Similarity Metrics for Markov Decision Processes [62.997667081978825]
  In recent years, transfer learning has succeeded in making Reinforcement Learning (RL) algorithms more efficient. In this paper, we propose a categorization of these metrics and analyze the definitions of similarity proposed so far.
  arXiv  Detail & Related papers  (2021-03-08T12:36:42Z)
• Meta-Baseline: Exploring Simple Meta-Learning for Few-Shot Learning [79.25478727351604]
  We explore a simple process: meta-learning over a whole-classification pre-trained model on its evaluation metric. We observe this simple method achieves competitive performance to state-of-the-art methods on standard benchmarks.
  arXiv  Detail & Related papers  (2020-03-09T20:06:36Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.