CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP

• URL: http://arxiv.org/abs/2104.08835v1
• Date: Sun, 18 Apr 2021 12:14:46 GMT
• Title: CrossFit: A Few-shot Learning Challenge for Cross-task Generalization in NLP
• Authors: Qinyuan Ye, Bill Yuchen Lin, Xiang Ren
• Abstract summary: We introduce CrossFit, a task setup for studying cross-task few-shot learning ability. We present NLP Few-shot Gym, a repository of 160 few-shot NLP tasks. Our empirical analysis reveals that the few-shot learning ability on unseen tasks can be improved via an upstream learning stage.
• Score: 38.40614678878222
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Humans can learn a new language task more efficiently than machines, conceivably by leveraging their prior experience and knowledge in learning other tasks. In this paper, we explore whether such cross-task generalization ability can be acquired, and further applied to build better few-shot learners across diverse NLP tasks. We introduce CrossFit, a task setup for studying cross-task few-shot learning ability, which standardizes seen/unseen task splits, data access during different learning stages, and the evaluation protocols. In addition, we present NLP Few-shot Gym, a repository of 160 few-shot NLP tasks, covering diverse task categories and applications, and converted to a unified text-to-text format. Our empirical analysis reveals that the few-shot learning ability on unseen tasks can be improved via an upstream learning stage using a set of seen tasks. Additionally, the advantage lasts into medium-resource scenarios when thousands of training examples are available. We also observe that selection of upstream learning tasks can significantly influence few-shot performance on unseen tasks, asking further analysis on task similarity and transferability.

Related papers

• Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond [62.406687088097605]
  Multi-Task Learning (MTL) is a framework, where multiple related tasks are learned jointly and benefit from a shared representation space. We show that MTL can be successful with classification tasks with little, or non-overlapping annotations. We propose a novel approach, where knowledge exchange is enabled between the tasks via distribution matching.
  arXiv  Detail & Related papers  (2024-01-02T14:18:11Z)
• Object-Centric Multi-Task Learning for Human Instances [8.035105819936808]
  We explore a compact multi-task network architecture that maximally shares the parameters of the multiple tasks via object-centric learning. We propose a novel query design to encode the human instance information effectively, called human-centric query (HCQ) Experimental results show that the proposed multi-task network achieves comparable accuracy to state-of-the-art task-specific models.
  arXiv  Detail & Related papers  (2023-03-13T01:10:50Z)
• Active Task Randomization: Learning Robust Skills via Unsupervised Generation of Diverse and Feasible Tasks [37.73239471412444]
  We introduce Active Task Randomization (ATR), an approach that learns robust skills through the unsupervised generation of training tasks. ATR selects suitable tasks, which consist of an initial environment state and manipulation goal, for learning robust skills by balancing the diversity and feasibility of the tasks. We demonstrate that the learned skills can be composed by a task planner to solve unseen sequential manipulation problems based on visual inputs.
  arXiv  Detail & Related papers  (2022-11-11T11:24:55Z)
• Task Compass: Scaling Multi-task Pre-training with Task Prefix [122.49242976184617]
  Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks. We propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks. Our model can not only serve as the strong foundation backbone for a wide range of tasks but also be feasible as a probing tool for analyzing task relationships.
  arXiv  Detail & Related papers  (2022-10-12T15:02:04Z)
• Saliency-Regularized Deep Multi-Task Learning [7.3810864598379755]
  Multitask learning enforces multiple learning tasks to share knowledge to improve their generalization abilities. Modern deep multitask learning can jointly learn latent features and task sharing, but they are obscure in task relation. This paper proposes a new multitask learning framework that jointly learns latent features and explicit task relations.
  arXiv  Detail & Related papers  (2022-07-03T20:26:44Z)
• Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning [28.104054292437525]
  We disentangle the effect of scale and relatedness of tasks in multi-task representation learning. If the target tasks are known ahead of time, then training on a smaller set of related tasks is competitive to the large-scale multi-task training.
  arXiv  Detail & Related papers  (2022-04-23T18:11:35Z)
• A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods [17.094426577723507]
  Multi-task learning (MTL) has become increasingly popular in natural language processing (NLP) It improves the performance of related tasks by exploiting their commonalities and differences. It is still not understood very well how multi-task learning can be implemented based on the relatedness of training tasks.
  arXiv  Detail & Related papers  (2022-04-07T15:22:19Z)
• On Steering Multi-Annotations per Sample for Multi-Task Learning [79.98259057711044]
  The study of multi-task learning has drawn great attention from the community. Despite the remarkable progress, the challenge of optimally learning different tasks simultaneously remains to be explored. Previous works attempt to modify the gradients from different tasks. Yet these methods give a subjective assumption of the relationship between tasks, and the modified gradient may be less accurate. In this paper, we introduce Task Allocation(STA), a mechanism that addresses this issue by a task allocation approach, in which each sample is randomly allocated a subset of tasks. For further progress, we propose Interleaved Task Allocation(ISTA) to iteratively allocate all
  arXiv  Detail & Related papers  (2022-03-06T11:57:18Z)
• Variational Multi-Task Learning with Gumbel-Softmax Priors [105.22406384964144]
  Multi-task learning aims to explore task relatedness to improve individual tasks. We propose variational multi-task learning (VMTL), a general probabilistic inference framework for learning multiple related tasks.
  arXiv  Detail & Related papers  (2021-11-09T18:49:45Z)
• Exploring and Predicting Transferability across NLP Tasks [115.6278033699853]
  We study the transferability between 33 NLP tasks across three broad classes of problems. Our results show that transfer learning is more beneficial than previously thought. We also develop task embeddings that can be used to predict the most transferable source tasks for a given target task.
  arXiv  Detail & Related papers  (2020-05-02T09:39:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.