Do Text-to-Text Multi-Task Learners Suffer from Task Conflict?

• URL: http://arxiv.org/abs/2212.06645v1
• Date: Tue, 13 Dec 2022 15:28:57 GMT
• Title: Do Text-to-Text Multi-Task Learners Suffer from Task Conflict?
• Abstract summary: A new type of multi-task learning within NLP homogenizes multi-task architectures as a shared encoder and language model decoder. We study how certain factors in the shift towards text-to-text models affects multi-task conflict and negative transfer.
• Score: 17.861704408540444
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Traditional multi-task learning architectures train a single model across multiple tasks through a shared encoder followed by task-specific decoders. Learning these models often requires specialized training algorithms that address task-conflict in the shared parameter updates, which otherwise can lead to negative transfer. A new type of multi-task learning within NLP homogenizes multi-task architectures as a shared encoder and language model decoder, which does surprisingly well across a range of diverse tasks. Does this new architecture suffer from task-conflicts that require specialized training algorithms? We study how certain factors in the shift towards text-to-text models affects multi-task conflict and negative transfer, finding that both directional conflict and transfer are surprisingly constant across architectures.

Related papers

• Unified Multi-Task Relevance Modeling for E-Commerce: Comparing Task Routing Architectures Across LLMs and Cross-Encoders [5.423726451729119]
  Current industry practice relies on separate models for each task, preventing knowledge transfer and producing inconsistent relevance signals.<n>Our work is driven by the following insight: encoder based and decoder only models encode task identity through different mechanisms.<n>We propose a unified multi task framework that jointly trains on six entity pair tasks under a shared three point relevance scale.
  arXiv  Detail & Related papers  (2026-06-22T20:32:47Z)
• RepVF: A Unified Vector Fields Representation for Multi-task 3D Perception [64.80760846124858]
  This paper proposes a novel unified representation, RepVF, which harmonizes the representation of various perception tasks. RepVF characterizes the structure of different targets in the scene through a vector field, enabling a single-head, multi-task learning model. Building upon RepVF, we introduce RFTR, a network designed to exploit the inherent connections between different tasks.
  arXiv  Detail & Related papers  (2024-07-15T16:25:07Z)
• YOLOR-Based Multi-Task Learning [12.5920336941241]
  Multi-task learning (MTL) aims to learn multiple tasks using a single model and jointly improve all of them assuming generalization and shared semantics. We propose building on You Only Learn One Representation (YOLOR), a network architecture specifically designed for multitasking. We find that our method achieves competitive performance on all tasks while maintaining a low parameter count and without any pre-training.
  arXiv  Detail & Related papers  (2023-09-29T01:42:21Z)
• TaskExpert: Dynamically Assembling Multi-Task Representations with Memorial Mixture-of-Experts [11.608682595506354]
  Recent models consider directly decoding task-specific features from one shared task-generic feature. As the input feature is fully shared and each task decoder also shares decoding parameters for different input samples, it leads to a static feature decoding process. We propose TaskExpert, a novel multi-task mixture-of-experts model that enables learning multiple representative task-generic feature spaces.
  arXiv  Detail & Related papers  (2023-07-28T06:00:57Z)
• JiuZhang 2.0: A Unified Chinese Pre-trained Language Model for Multi-task Mathematical Problem Solving [77.51817534090789]
  We propose textbfJiuZhang2.0, a unified Chinese PLM specially for multi-task mathematical problem solving. Our idea is to maintain a moderate-sized model and employ the emphcross-task knowledge sharing to improve the model capacity in a multi-task setting.
  arXiv  Detail & Related papers  (2023-06-19T15:45:36Z)
• Improving Cross-task Generalization of Unified Table-to-text Models with Compositional Task Configurations [63.04466647849211]
  Methods typically encode task information with a simple dataset name as a prefix to the encoder. We propose compositional task configurations, a set of prompts prepended to the encoder to improve cross-task generalization. We show this not only allows the model to better learn shared knowledge across different tasks at training, but also allows us to control the model by composing new configurations.
  arXiv  Detail & Related papers  (2022-12-17T02:20:14Z)
• Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners [67.5865966762559]
  We study whether sparsely activated Mixture-of-Experts (MoE) improve multi-task learning. We devise task-aware gating functions to route examples from different tasks to specialized experts. This results in a sparsely activated multi-task model with a large number of parameters, but with the same computational cost as that of a dense model.
  arXiv  Detail & Related papers  (2022-04-16T00:56:12Z)
• Multi-Task Learning with Deep Neural Networks: A Survey [0.0]
  Multi-task learning (MTL) is a subfield of machine learning in which multiple tasks are simultaneously learned by a shared model. We give an overview of multi-task learning methods for deep neural networks, with the aim of summarizing both the well-established and most recent directions within the field.
  arXiv  Detail & Related papers  (2020-09-10T19:31:04Z)
• Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference [75.95287293847697]
  Two common challenges in developing multi-task models are often overlooked in literature. First, enabling the model to be inherently incremental, continuously incorporating information from new tasks without forgetting the previously learned ones (incremental learning) Second, eliminating adverse interactions amongst tasks, which has been shown to significantly degrade the single-task performance in a multi-task setup (task interference)
  arXiv  Detail & Related papers  (2020-07-24T14:44:46Z)
• Navigating the Trade-Off between Multi-Task Learning and Learning to Multitask in Deep Neural Networks [9.278739724750343]
  Multi-task learning refers to a paradigm in machine learning in which a network is trained on various related tasks to facilitate the acquisition of tasks. multitasking is used to indicate, especially in the cognitive science literature, the ability to execute multiple tasks simultaneously. We show that the same tension arises in deep networks and discuss a meta-learning algorithm for an agent to manage this trade-off in an unfamiliar environment.
  arXiv  Detail & Related papers  (2020-07-20T23:26:16Z)
• Adversarial Continual Learning [99.56738010842301]
  We propose a hybrid continual learning framework that learns a disjoint representation for task-invariant and task-specific features. Our model combines architecture growth to prevent forgetting of task-specific skills and an experience replay approach to preserve shared skills.
  arXiv  Detail & Related papers  (2020-03-21T02:08:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.