Task Compass: Scaling Multi-task Pre-training with Task Prefix

• URL: http://arxiv.org/abs/2210.06277v1
• Date: Wed, 12 Oct 2022 15:02:04 GMT
• Title: Task Compass: Scaling Multi-task Pre-training with Task Prefix
• Authors: Zhuosheng Zhang, Shuohang Wang, Yichong Xu, Yuwei Fang, Wenhao Yu, Yang Liu, Hai Zhao, Chenguang Zhu and Michael Zeng
• Abstract summary: Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks. We propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks. Our model can not only serve as the strong foundation backbone for a wide range of tasks but also be feasible as a probing tool for analyzing task relationships.
• Score: 122.49242976184617
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Leveraging task-aware annotated data as supervised signals to assist with self-supervised learning on large-scale unlabeled data has become a new trend in pre-training language models. Existing studies show that multi-task learning with large-scale supervised tasks suffers from negative effects across tasks. To tackle the challenge, we propose a task prefix guided multi-task pre-training framework to explore the relationships among tasks. We conduct extensive experiments on 40 datasets, which show that our model can not only serve as the strong foundation backbone for a wide range of tasks but also be feasible as a probing tool for analyzing task relationships. The task relationships reflected by the prefixes align transfer learning performance between tasks. They also suggest directions for data augmentation with complementary tasks, which help our model achieve human-parity results on commonsense reasoning leaderboards. Code is available at https://github.com/cooelf/CompassMTL

Related papers

• Data-CUBE: Data Curriculum for Instruction-based Sentence Representation Learning [85.66907881270785]
  We propose a data curriculum method, namely Data-CUBE, that arranges the orders of all the multi-task data for training. In the task level, we aim to find the optimal task order to minimize the total cross-task interference risk. In the instance level, we measure the difficulty of all instances per task, then divide them into the easy-to-difficult mini-batches for training.
  arXiv  Detail & Related papers  (2024-01-07T18:12:20Z)
• Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond [62.406687088097605]
  Multi-Task Learning (MTL) is a framework, where multiple related tasks are learned jointly and benefit from a shared representation space. We show that MTL can be successful with classification tasks with little, or non-overlapping annotations. We propose a novel approach, where knowledge exchange is enabled between the tasks via distribution matching.
  arXiv  Detail & Related papers  (2024-01-02T14:18:11Z)
• Identification of Negative Transfers in Multitask Learning Using Surrogate Models [29.882265735630046]
  Multitask learning is widely used to train a low-resource target task by augmenting it with multiple related source tasks. A critical problem in multitask learning is identifying subsets of source tasks that would benefit the target task. We introduce an efficient procedure to address this problem via surrogate modeling.
  arXiv  Detail & Related papers  (2023-03-25T23:16:11Z)
• TaskMix: Data Augmentation for Meta-Learning of Spoken Intent Understanding [0.0]
  We show that a state-of-the-art data augmentation method worsens this problem of overfitting when the task diversity is low. We propose a simple method, TaskMix, which synthesizes new tasks by linearly interpolating existing tasks. We show that TaskMix outperforms baselines, alleviates overfitting when task diversity is low, and does not degrade performance even when it is high.
  arXiv  Detail & Related papers  (2022-09-26T00:37:40Z)
• Fast Inference and Transfer of Compositional Task Structures for Few-shot Task Generalization [101.72755769194677]
  We formulate it as a few-shot reinforcement learning problem where a task is characterized by a subtask graph. Our multi-task subtask graph inferencer (MTSGI) first infers the common high-level task structure in terms of the subtask graph from the training tasks. Our experiment results on 2D grid-world and complex web navigation domains show that the proposed method can learn and leverage the common underlying structure of the tasks for faster adaptation to the unseen tasks.
  arXiv  Detail & Related papers  (2022-05-25T10:44:25Z)
• Learning Multiple Dense Prediction Tasks from Partially Annotated Data [41.821234589075445]
  We look at jointly learning of multiple dense prediction tasks on partially annotated data, which we call multi-task partially-supervised learning. We propose a multi-task training procedure that successfully leverages task relations to supervise its multi-task learning when data is partially annotated. We rigorously demonstrate that our proposed method effectively exploits the images with unlabelled tasks and outperforms existing semi-supervised learning approaches and related methods on three standard benchmarks.
  arXiv  Detail & Related papers  (2021-11-29T19:03:12Z)
• Efficiently Identifying Task Groupings for Multi-Task Learning [55.80489920205404]
  Multi-task learning can leverage information learned by one task to benefit the training of other tasks. We suggest an approach to select which tasks should train together in multi-task learning models. Our method determines task groupings in a single training run by co-training all tasks together and quantifying the effect to which one task's gradient would affect another task's loss.
  arXiv  Detail & Related papers  (2021-09-10T02:01:43Z)
• Temporally Correlated Task Scheduling for Sequence Learning [143.70523777803723]
  In many applications, a sequence learning task is usually associated with multiple temporally correlated auxiliary tasks. We introduce a learnable scheduler to sequence learning, which can adaptively select auxiliary tasks for training. Our method significantly improves the performance of simultaneous machine translation and stock trend forecasting.
  arXiv  Detail & Related papers  (2020-07-10T10:28:54Z)
• Understanding and Improving Information Transfer in Multi-Task Learning [14.43111978531182]
  We study an architecture with a shared module for all tasks and a separate output module for each task. We show that misalignment between task data can cause negative transfer (or hurt performance) and provide sufficient conditions for positive transfer. Inspired by the theoretical insights, we show that aligning tasks' embedding layers leads to performance gains for multi-task training and transfer learning.
  arXiv  Detail & Related papers  (2020-05-02T23:43:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.