Task Attended Meta-Learning for Few-Shot Learning
- URL: http://arxiv.org/abs/2106.10642v1
- Date: Sun, 20 Jun 2021 07:34:37 GMT
- Title: Task Attended Meta-Learning for Few-Shot Learning
- Authors: Aroof Aimen, Sahil Sidheekh, Narayanan C. Krishnan
- Abstract summary: We introduce a training curriculum motivated by selective focus in humans, called task attended meta-training, to weight the tasks in a batch.
The comparisons of the models with their non-task-attended counterparts on complex datasets validate its effectiveness.
- Score: 3.0724051098062097
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Meta-learning (ML) has emerged as a promising direction in learning models
under constrained resource settings like few-shot learning. The popular
approaches for ML either learn a generalizable initial model or a generic
parametric optimizer through episodic training. The former approaches leverage
the knowledge from a batch of tasks to learn an optimal prior. In this work, we
study the importance of a batch for ML. Specifically, we first incorporate a
batch episodic training regimen to improve the learning of the generic
parametric optimizer. We also hypothesize that the common assumption in batch
episodic training that each task in a batch has an equal contribution to
learning an optimal meta-model need not be true. We propose to weight the tasks
in a batch according to their "importance" in improving the meta-model's
learning. To this end, we introduce a training curriculum motivated by
selective focus in humans, called task attended meta-training, to weight the
tasks in a batch. Task attention is a standalone module that can be integrated
with any batch episodic training regimen. The comparisons of the models with
their non-task-attended counterparts on complex datasets like miniImageNet and
tieredImageNet validate its effectiveness.
Related papers
- $α$VIL: Learning to Leverage Auxiliary Tasks for Multitask Learning [3.809702129519642]
Multitask Learning aims to train a range of (usually related) tasks with the help of a shared model.
It becomes important to estimate the positive or negative influence auxiliary tasks will have on the target.
We propose a novel method called $alpha$Variable Learning ($alpha$VIL) that is able to adjust task weights dynamically during model training.
arXiv Detail & Related papers (2024-05-13T14:12:33Z) - Architecture, Dataset and Model-Scale Agnostic Data-free Meta-Learning [119.70303730341938]
We propose ePisode cUrriculum inveRsion (ECI) during data-free meta training and invErsion calibRation following inner loop (ICFIL) during meta testing.
ECI adaptively increases the difficulty level of pseudo episodes according to the real-time feedback of the meta model.
We formulate the optimization process of meta training with ECI as an adversarial form in an end-to-end manner.
arXiv Detail & Related papers (2023-03-20T15:10:41Z) - Task Adaptive Parameter Sharing for Multi-Task Learning [114.80350786535952]
Adaptive Task Adapting Sharing (TAPS) is a method for tuning a base model to a new task by adaptively modifying a small, task-specific subset of layers.
Compared to other methods, TAPS retains high accuracy on downstream tasks while introducing few task-specific parameters.
We evaluate our method on a suite of fine-tuning tasks and architectures (ResNet, DenseNet, ViT) and show that it achieves state-of-the-art performance while being simple to implement.
arXiv Detail & Related papers (2022-03-30T23:16:07Z) - Curriculum Meta-Learning for Few-shot Classification [1.5039745292757671]
We propose an adaptation of the curriculum training framework, applicable to state-of-the-art meta learning techniques for few-shot classification.
Our experiments with the MAML algorithm on two few-shot image classification tasks show significant gains with the curriculum training framework.
arXiv Detail & Related papers (2021-12-06T10:29:23Z) - MetaICL: Learning to Learn In Context [87.23056864536613]
We introduce MetaICL, a new meta-training framework for few-shot learning where a pretrained language model is tuned to do in-context learn-ing on a large set of training tasks.
We show that MetaICL approaches (and sometimes beats) the performance of models fully finetuned on the target task training data, and outperforms much bigger models with nearly 8x parameters.
arXiv Detail & Related papers (2021-10-29T17:42:08Z) - Meta-Regularization by Enforcing Mutual-Exclusiveness [0.8057006406834467]
We propose a regularization technique for meta-learning models that gives the model designer more control over the information flow during meta-training.
Our proposed regularization function shows an accuracy boost of $sim$ $36%$ on the Omniglot dataset.
arXiv Detail & Related papers (2021-01-24T22:57:19Z) - Variable-Shot Adaptation for Online Meta-Learning [123.47725004094472]
We study the problem of learning new tasks from a small, fixed number of examples, by meta-learning across static data from a set of previous tasks.
We find that meta-learning solves the full task set with fewer overall labels and greater cumulative performance, compared to standard supervised methods.
These results suggest that meta-learning is an important ingredient for building learning systems that continuously learn and improve over a sequence of problems.
arXiv Detail & Related papers (2020-12-14T18:05:24Z) - Self-Supervised Meta-Learning for Few-Shot Natural Language
Classification Tasks [40.97125791174191]
We propose a self-supervised approach to generate a large, rich, meta-learning task distribution from unlabeled text.
We show that this meta-training leads to better few-shot generalization than language-model pre-training followed by finetuning.
arXiv Detail & Related papers (2020-09-17T17:53:59Z) - Improving Generalization in Meta-learning via Task Augmentation [69.83677015207527]
We propose two task augmentation methods, including MetaMix and Channel Shuffle.
Both MetaMix and Channel Shuffle outperform state-of-the-art results by a large margin across many datasets.
arXiv Detail & Related papers (2020-07-26T01:50:42Z) - Pre-training Text Representations as Meta Learning [113.3361289756749]
We introduce a learning algorithm which directly optimize model's ability to learn text representations for effective learning of downstream tasks.
We show that there is an intrinsic connection between multi-task pre-training and model-agnostic meta-learning with a sequence of meta-train steps.
arXiv Detail & Related papers (2020-04-12T09:05:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.