On Hard Episodes in Meta-Learning

• URL: http://arxiv.org/abs/2110.11190v1
• Date: Thu, 21 Oct 2021 14:58:58 GMT
• Title: On Hard Episodes in Meta-Learning
• Abstract summary: Existing meta-learners primarily focus on improving the average task accuracy across multiple episodes. Different episodes, however, may vary in hardness and quality leading to a wide gap in the meta-learner's performance across episodes.
• Score: 35.46163595112011
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Existing meta-learners primarily focus on improving the average task accuracy across multiple episodes. Different episodes, however, may vary in hardness and quality leading to a wide gap in the meta-learner's performance across episodes. Understanding this issue is particularly critical in industrial few-shot settings, where there is limited control over test episodes as they are typically uploaded by end-users. In this paper, we empirically analyse the behaviour of meta-learners on episodes of varying hardness across three standard benchmark datasets: CIFAR-FS, mini-ImageNet, and tiered-ImageNet. Surprisingly, we observe a wide gap in accuracy of around 50% between the hardest and easiest episodes across all the standard benchmarks and meta-learners. We additionally investigate various properties of hard episodes and highlight their connection to catastrophic forgetting during meta-training. To address the issue of sub-par performance on hard episodes, we investigate and benchmark different meta-training strategies based on adversarial training and curriculum learning. We find that adversarial training strategies are much more powerful than curriculum learning in improving the prediction performance on hard episodes.

Related papers

• Episodic-free Task Selection for Few-shot Learning [2.508902852545462]
  We propose a novel meta-training framework beyond episodic training. episodic tasks are not used directly for training, but for evaluating the effectiveness of some selected episodic-free tasks. In experiments, the training task set contains some promising types, e. g., contrastive learning and classification.
  arXiv  Detail & Related papers  (2024-01-31T10:52:15Z)
• Multi-Label Meta Weighting for Long-Tailed Dynamic Scene Graph Generation [55.429541407920304]
  Recognizing the predicate between subject and object pairs is imbalanced and multi-label in nature. Recent state-of-the-art methods predominantly focus on the most frequently occurring predicate classes. We introduce a multi-label meta-learning framework to deal with the biased predicate distribution.
  arXiv  Detail & Related papers  (2023-06-16T18:14:23Z)
• Uniform Sampling over Episode Difficulty [55.067544082168624]
  We propose a method to approximate episode sampling distributions based on their difficulty. As the proposed sampling method is algorithm agnostic, we can leverage these insights to improve few-shot learning accuracies.
  arXiv  Detail & Related papers  (2021-08-03T17:58:54Z)
• Variable-Shot Adaptation for Online Meta-Learning [123.47725004094472]
  We study the problem of learning new tasks from a small, fixed number of examples, by meta-learning across static data from a set of previous tasks. We find that meta-learning solves the full task set with fewer overall labels and greater cumulative performance, compared to standard supervised methods. These results suggest that meta-learning is an important ingredient for building learning systems that continuously learn and improve over a sequence of problems.
  arXiv  Detail & Related papers  (2020-12-14T18:05:24Z)
• Expert Training: Task Hardness Aware Meta-Learning for Few-Shot Classification [62.10696018098057]
  We propose an easy-to-hard expert meta-training strategy to arrange the training tasks properly. A task hardness aware module is designed and integrated into the training procedure to estimate the hardness of a task. Experimental results on the miniImageNet and tieredImageNetSketch datasets show that the meta-learners can obtain better results with our expert training strategy.
  arXiv  Detail & Related papers  (2020-07-13T08:49:00Z)
• Incremental Meta-Learning via Indirect Discriminant Alignment [118.61152684795178]
  We develop a notion of incremental learning during the meta-training phase of meta-learning. Our approach performs favorably at test time as compared to training a model with the full meta-training set.
  arXiv  Detail & Related papers  (2020-02-11T01:39:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.