Boosting a Model Zoo for Multi-Task and Continual Learning

• URL: http://arxiv.org/abs/2106.03027v1
• Date: Sun, 6 Jun 2021 04:25:09 GMT
• Title: Boosting a Model Zoo for Multi-Task and Continual Learning
• Authors: Rahul Ramesh, Pratik Chaudhari
• Abstract summary: "Model Zoo" is an algorithm that builds an ensemble of models, each of which is very small, and it is trained on a smaller set of tasks. Model Zoo achieves large gains in prediction accuracy compared to state-of-the-art methods in multi-task and continual learning.
• Score: 15.110807414130923
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Leveraging data from multiple tasks, either all at once, or incrementally, to learn one model is an idea that lies at the heart of multi-task and continual learning methods. Ideally, such a model predicts each task more accurately than if the task were trained in isolation. We show using tools in statistical learning theory (i) how tasks can compete for capacity, i.e., including a particular task can deteriorate the accuracy on a given task, and (ii) that the ideal set of tasks that one should train together in order to perform well on a given task is different for different tasks. We develop methods to discover such competition in typical benchmark datasets which suggests that the prevalent practice of training with all tasks leaves performance on the table. This motivates our "Model Zoo", which is a boosting-based algorithm that builds an ensemble of models, each of which is very small, and it is trained on a smaller set of tasks. Model Zoo achieves large gains in prediction accuracy compared to state-of-the-art methods across a variety of existing benchmarks in multi-task and continual learning, as well as more challenging ones of our creation. We also show that even a model trained independently on all tasks outperforms all existing multi-task and continual learning methods.

Related papers

• Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond [62.406687088097605]
  Multi-Task Learning (MTL) is a framework, where multiple related tasks are learned jointly and benefit from a shared representation space. We show that MTL can be successful with classification tasks with little, or non-overlapping annotations. We propose a novel approach, where knowledge exchange is enabled between the tasks via distribution matching.
  arXiv  Detail & Related papers  (2024-01-02T14:18:11Z)
• Exploring intra-task relations to improve meta-learning algorithms [1.223779595809275]
  We aim to exploit external knowledge of task relations to improve training stability via effective mini-batching of tasks. We hypothesize that selecting a diverse set of tasks in a mini-batch will lead to a better estimate of the full gradient and hence will lead to a reduction of noise in training.
  arXiv  Detail & Related papers  (2023-12-27T15:33:52Z)
• An Efficient General-Purpose Modular Vision Model via Multi-Task Heterogeneous Training [79.78201886156513]
  We present a model that can perform multiple vision tasks and can be adapted to other downstream tasks efficiently. Our approach achieves comparable results to single-task state-of-the-art models and demonstrates strong generalization on downstream tasks.
  arXiv  Detail & Related papers  (2023-06-29T17:59:57Z)
• Identification of Negative Transfers in Multitask Learning Using Surrogate Models [29.882265735630046]
  Multitask learning is widely used to train a low-resource target task by augmenting it with multiple related source tasks. A critical problem in multitask learning is identifying subsets of source tasks that would benefit the target task. We introduce an efficient procedure to address this problem via surrogate modeling.
  arXiv  Detail & Related papers  (2023-03-25T23:16:11Z)
• Task Compass: Scaling Multi-task Pre-training with Task Prefix [122.49242976184617]
  Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks. We propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks. Our model can not only serve as the strong foundation backbone for a wide range of tasks but also be feasible as a probing tool for analyzing task relationships.
  arXiv  Detail & Related papers  (2022-10-12T15:02:04Z)
• An Evolutionary Approach to Dynamic Introduction of Tasks in Large-scale Multitask Learning Systems [4.675744559395732]
  Multitask learning assumes that models capable of learning from multiple tasks can achieve better quality and efficiency via knowledge transfer. State of the art ML models rely on high customization for each task and leverage size and data scale rather than scaling the number of tasks. We propose an evolutionary method that can generate a large scale multitask model and can support the dynamic and continuous addition of new tasks.
  arXiv  Detail & Related papers  (2022-05-25T13:10:47Z)
• Explaining the Effectiveness of Multi-Task Learning for Efficient Knowledge Extraction from Spine MRI Reports [2.5953185061765884]
  We show that a single multi-tasking model can match the performance of task specific models. We validate our observations on our internal radiologist-annotated datasets on the cervical and lumbar spine.
  arXiv  Detail & Related papers  (2022-05-06T01:51:19Z)
• The Effect of Diversity in Meta-Learning [79.56118674435844]
  Few-shot learning aims to learn representations that can tackle novel tasks given a small number of examples. Recent studies show that task distribution plays a vital role in the model's performance. We study different task distributions on a myriad of models and datasets to evaluate the effect of task diversity on meta-learning algorithms.
  arXiv  Detail & Related papers  (2022-01-27T19:39:07Z)
• Variational Multi-Task Learning with Gumbel-Softmax Priors [105.22406384964144]
  Multi-task learning aims to explore task relatedness to improve individual tasks. We propose variational multi-task learning (VMTL), a general probabilistic inference framework for learning multiple related tasks.
  arXiv  Detail & Related papers  (2021-11-09T18:49:45Z)
• Efficiently Identifying Task Groupings for Multi-Task Learning [55.80489920205404]
  Multi-task learning can leverage information learned by one task to benefit the training of other tasks. We suggest an approach to select which tasks should train together in multi-task learning models. Our method determines task groupings in a single training run by co-training all tasks together and quantifying the effect to which one task's gradient would affect another task's loss.
  arXiv  Detail & Related papers  (2021-09-10T02:01:43Z)
• Knowledge Distillation for Multi-task Learning [38.20005345733544]
  Multi-task learning (MTL) is to learn one single model that performs multiple tasks for achieving good performance on all tasks and lower cost on computation. Learning such a model requires to jointly optimize losses of a set of tasks with different difficulty levels, magnitudes, and characteristics. We propose a knowledge distillation based method in this work to address the imbalance problem in multi-task learning.
  arXiv  Detail & Related papers  (2020-07-14T08:02:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.