Learning Boost by Exploiting the Auxiliary Task in Multi-task Domain

• URL: http://arxiv.org/abs/2008.02043v1
• Date: Wed, 5 Aug 2020 10:56:56 GMT
• Title: Learning Boost by Exploiting the Auxiliary Task in Multi-task Domain
• Authors: Jonghwa Yim, Sang Hwan Kim
• Abstract summary: Learning two tasks in a single shared function has some benefits. It helps to generalize the function that can be learned using generally applicable information for both tasks. However, in a real environment, tasks inevitably have a conflict between them during the learning phase, called negative transfer. We introduce a novel approach that can drive positive transfer and suppress negative transfer by leveraging class-wise weights in the learning process.
• Score: 1.2183405753834562
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Learning two tasks in a single shared function has some benefits. Firstly by acquiring information from the second task, the shared function leverages useful information that could have been neglected or underestimated in the first task. Secondly, it helps to generalize the function that can be learned using generally applicable information for both tasks. To fully enjoy these benefits, Multi-task Learning (MTL) has long been researched in various domains such as computer vision, language understanding, and speech synthesis. While MTL benefits from the positive transfer of information from multiple tasks, in a real environment, tasks inevitably have a conflict between them during the learning phase, called negative transfer. The negative transfer hampers function from achieving the optimality and degrades the performance. To solve the problem of the task conflict, previous works only suggested partial solutions that are not fundamental, but ad-hoc. A common approach is using a weighted sum of losses. The weights are adjusted to induce positive transfer. Paradoxically, this kind of solution acknowledges the problem of negative transfer and cannot remove it unless the weight of the task is set to zero. Therefore, these previous methods had limited success. In this paper, we introduce a novel approach that can drive positive transfer and suppress negative transfer by leveraging class-wise weights in the learning process. The weights act as an arbitrator of the fundamental unit of information to determine its positive or negative status to the main task.

Related papers

• Enabling Asymmetric Knowledge Transfer in Multi-Task Learning with Self-Auxiliaries [4.031100721019478]
  We investigate asymmetric task relationships, where knowledge transfer aids the learning of certain tasks while hindering the learning of others. We propose an optimisation strategy that includes additional cloned tasks named self-auxiliaries into the learning process to flexibly transfer knowledge between tasks asymmetrically. We demonstrate that asymmetric knowledge transfer provides substantial improvements in performance compared to existing multi-task optimisation strategies on benchmark computer vision problems.
  arXiv  Detail & Related papers  (2024-10-21T10:57:25Z)
• Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond [62.406687088097605]
  Multi-Task Learning (MTL) is a framework, where multiple related tasks are learned jointly and benefit from a shared representation space. We show that MTL can be successful with classification tasks with little, or non-overlapping annotations. We propose a novel approach, where knowledge exchange is enabled between the tasks via distribution matching.
  arXiv  Detail & Related papers  (2024-01-02T14:18:11Z)
• Mitigating Negative Transfer with Task Awareness for Sexism, Hate Speech, and Toxic Language Detection [7.661927086611542]
  This paper proposes a new approach to mitigate the negative transfer problem based on the task awareness concept. The proposed approach results in diminishing the negative transfer together with an improvement of performance over classic MTL solution. The proposed approach has been implemented in two unified architectures to detect Sexism, Hate Speech, and Toxic Language in text comments.
  arXiv  Detail & Related papers  (2023-07-07T04:10:37Z)
• ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning [59.08197876733052]
  Auxiliary-Task Learning (ATL) aims to improve the performance of the target task by leveraging the knowledge obtained from related tasks. Sometimes, learning multiple tasks simultaneously results in lower accuracy than learning only the target task, known as negative transfer. ForkMerge is a novel approach that periodically forks the model into multiple branches, automatically searches the varying task weights.
  arXiv  Detail & Related papers  (2023-01-30T02:27:02Z)
• Multi-task Bias-Variance Trade-off Through Functional Constraints [102.64082402388192]
  Multi-task learning aims to acquire a set of functions that perform well for diverse tasks. In this paper we draw intuition from the two extreme learning scenarios -- a single function for all tasks, and a task-specific function that ignores the other tasks. We introduce a constrained learning formulation that enforces domain specific solutions to a central function.
  arXiv  Detail & Related papers  (2022-10-27T16:06:47Z)
• Identifying Suitable Tasks for Inductive Transfer Through the Analysis of Feature Attributions [78.55044112903148]
  We use explainability techniques to predict whether task pairs will be complementary, through comparison of neural network activation between single-task models. Our results show that, through this approach, it is possible to reduce training time by up to 83.5% at a cost of only 0.034 reduction in positive-class F1 on the TREC-IS 2020-A dataset.
  arXiv  Detail & Related papers  (2022-02-02T15:51:07Z)
• Phase Transitions in Transfer Learning for High-Dimensional Perceptrons [12.614901374282868]
  Transfer learning seeks to improve the generalization performance of a target task by exploiting knowledge learned from a related source task. The latter question is related to the so-called negative transfer phenomenon, where the transferred source information actually reduces the generalization performance of the target task. We present a theoretical analysis of transfer learning by studying a pair of related perceptron learning tasks.
  arXiv  Detail & Related papers  (2021-01-06T08:29:22Z)
• Exploring and Predicting Transferability across NLP Tasks [115.6278033699853]
  We study the transferability between 33 NLP tasks across three broad classes of problems. Our results show that transfer learning is more beneficial than previously thought. We also develop task embeddings that can be used to predict the most transferable source tasks for a given target task.
  arXiv  Detail & Related papers  (2020-05-02T09:39:36Z)
• Task-Feature Collaborative Learning with Application to Personalized Attribute Prediction [166.87111665908333]
  We propose a novel multi-task learning method called Task-Feature Collaborative Learning (TFCL) Specifically, we first propose a base model with a heterogeneous block-diagonal structure regularizer to leverage the collaborative grouping of features and tasks. As a practical extension, we extend the base model by allowing overlapping features and differentiating the hard tasks.
  arXiv  Detail & Related papers  (2020-04-29T02:32:04Z)
• Representation Ensembling for Synergistic Lifelong Learning with Quasilinear Complexity [17.858926093389737]
  In lifelong learning, data are used to improve performance not only on the current task, but also on previously encountered, and as yet unencountered tasks. Our key insight is that we can synergistically ensemble representations -- that were learned independently on disparate tasks -- to enable both forward and backward transfer.
  arXiv  Detail & Related papers  (2020-04-27T16:16:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.