MT2ST: Adaptive Multi-Task to Single-Task Learning

• URL: http://arxiv.org/abs/2406.18038v2
• Date: Wed, 06 Nov 2024 10:14:20 GMT
• Title: MT2ST: Adaptive Multi-Task to Single-Task Learning
• Authors: Dong Liu,
• Abstract summary: Multi-Task to Single-Task (MT2ST) is a novel approach that significantly enhances the efficiency and accuracy of word embedding training. Our empirical studies demonstrate that MT2ST can reduce training time by 67% when contrasted with single-task learning approaches.
• Score: 7.307436175842646
• License:
• Abstract: The conventional training approaches often face challenges in balancing the breadth of multi-task learning (MTL) with the depth of single-task learning (STL). To address this issue, we introduce the Multi-Task to Single-Task (MT2ST) framework, a groundbreaking approach that can combine the generalizability of MTL with the precision of STL. Our work include two strategies: 'Diminish' and 'Switch'. 'Diminish' Strategy will gradually reduce the influence of auxiliary tasks, while the 'Switch' strategy involves a shift from multi-tasking to single-tasking at a specific timepoint at the training process. In this paper, we propose the Multi-Task to Single-Task (MT2ST) framework, a novel approach that significantly enhances the efficiency and accuracy of word embedding training while concurrently addressing prevalent issues such as overfitting. Our empirical studies demonstrate that MT2ST can reduce training time by 67% when contrasted with single-task learning approaches, and by 13% compared to traditional multi-task learning methods. These findings underscore MT2ST's potential to be a powerful tools for word embedding training acceleration. The code implementation is can be found at: https://github.com/NoakLiu/MT2ST-Word-Embeddings-Acceleration.

Related papers

• Cross-Task Affinity Learning for Multitask Dense Scene Predictions [5.939164722752263]
  Multitask learning (MTL) has become prominent for its ability to predict multiple tasks jointly. We introduce the Cross-Task Affinity Learning (CTAL) module, a lightweight framework that enhances task refinement in multitask networks. Our results demonstrate state-of-the-art MTL performance for both CNN and transformer backbones, using significantly fewer parameters than single-task learning.
  arXiv  Detail & Related papers  (2024-01-20T05:31:47Z)
• When Multi-Task Learning Meets Partial Supervision: A Computer Vision Review [7.776434991976473]
  Multi-Task Learning (MTL) aims to learn multiple tasks simultaneously while exploiting their mutual relationships. This review focuses on how MTL could be utilised under different partial supervision settings to address these challenges.
  arXiv  Detail & Related papers  (2023-07-25T20:08:41Z)
• M$^3$ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task Learning with Model-Accelerator Co-design [95.41238363769892]
  Multi-task learning (MTL) encapsulates multiple learned tasks in a single model and often lets those tasks learn better jointly. Current MTL regimes have to activate nearly the entire model even to just execute a single task. We present a model-accelerator co-design framework to enable efficient on-device MTL.
  arXiv  Detail & Related papers  (2022-10-26T15:40:24Z)
• Multi-Task Meta Learning: learn how to adapt to unseen tasks [4.287114092271669]
  This work proposes Multi-task Meta Learning (MTML), integrating two learning paradigms Multi-Task Learning (MTL) and meta learning. The fundamental idea is to train a multi-task model, such that when an unseen task is introduced, it can learn in fewer steps whilst offering a performance at least as good as conventional single task learning. MTML achieves state-of-the-art results for three out of four tasks for the NYU-v2 dataset and two out of four for the taskonomy dataset.
  arXiv  Detail & Related papers  (2022-10-13T12:59:54Z)
• Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners [67.5865966762559]
  We study whether sparsely activated Mixture-of-Experts (MoE) improve multi-task learning. We devise task-aware gating functions to route examples from different tasks to specialized experts. This results in a sparsely activated multi-task model with a large number of parameters, but with the same computational cost as that of a dense model.
  arXiv  Detail & Related papers  (2022-04-16T00:56:12Z)
• A Survey of Multi-task Learning in Natural Language Processing: Regarding Task Relatedness and Training Methods [17.094426577723507]
  Multi-task learning (MTL) has become increasingly popular in natural language processing (NLP) It improves the performance of related tasks by exploiting their commonalities and differences. It is still not understood very well how multi-task learning can be implemented based on the relatedness of training tasks.
  arXiv  Detail & Related papers  (2022-04-07T15:22:19Z)
• In Defense of the Unitary Scalarization for Deep Multi-Task Learning [121.76421174107463]
  We present a theoretical analysis suggesting that many specialized multi-tasks can be interpreted as forms of regularization. We show that, when coupled with standard regularization and stabilization techniques, unitary scalarization matches or improves upon the performance of complex multitasks.
  arXiv  Detail & Related papers  (2022-01-11T18:44:17Z)
• Measuring and Harnessing Transference in Multi-Task Learning [58.48659733262734]
  Multi-task learning can leverage information learned by one task to benefit the training of other tasks. We analyze the dynamics of information transfer, or transference, across tasks throughout training.
  arXiv  Detail & Related papers  (2020-10-29T08:25:43Z)
• Multi-Task Learning with Deep Neural Networks: A Survey [0.0]
  Multi-task learning (MTL) is a subfield of machine learning in which multiple tasks are simultaneously learned by a shared model. We give an overview of multi-task learning methods for deep neural networks, with the aim of summarizing both the well-established and most recent directions within the field.
  arXiv  Detail & Related papers  (2020-09-10T19:31:04Z)
• Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference [75.95287293847697]
  Two common challenges in developing multi-task models are often overlooked in literature. First, enabling the model to be inherently incremental, continuously incorporating information from new tasks without forgetting the previously learned ones (incremental learning) Second, eliminating adverse interactions amongst tasks, which has been shown to significantly degrade the single-task performance in a multi-task setup (task interference)
  arXiv  Detail & Related papers  (2020-07-24T14:44:46Z)
• Learning to Multi-Task Learn for Better Neural Machine Translation [53.06405021125476]
  Multi-task learning is an elegant approach to inject linguistic-related biases into neural machine translation models. We propose a novel framework for learning the training schedule, ie learning to multi-task learn, for the biased-MTL setting of interest. Experiments show the resulting automatically learned training schedulers are competitive with the best, and lead to up to +1.1 BLEU score improvements.
  arXiv  Detail & Related papers  (2020-01-10T03:12:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.