Boosting Share Routing for Multi-task Learning

• URL: http://arxiv.org/abs/2009.00387v2
• Date: Mon, 1 Mar 2021 12:00:31 GMT
• Title: Boosting Share Routing for Multi-task Learning
• Authors: Xiaokai Chen and Xiaoguang Gu and Libo Fu
• Abstract summary: Multi-task learning (MTL) aims to make full use of the knowledge contained in multi-task supervision signals to improve the overall performance. How to make the knowledge of multiple tasks shared appropriately is an open problem for MTL. We propose a general framework called Multi-Task Neural Architecture Search (MTNAS) to efficiently find a suitable sharing route for a given MTL problem.
• Score: 0.12891210250935145
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-task learning (MTL) aims to make full use of the knowledge contained in multi-task supervision signals to improve the overall performance. How to make the knowledge of multiple tasks shared appropriately is an open problem for MTL. Most existing deep MTL models are based on parameter sharing. However, suitable sharing mechanism is hard to design as the relationship among tasks is complicated. In this paper, we propose a general framework called Multi-Task Neural Architecture Search (MTNAS) to efficiently find a suitable sharing route for a given MTL problem. MTNAS modularizes the sharing part into multiple layers of sub-networks. It allows sparse connection among these sub-networks and soft sharing based on gating is enabled for a certain route. Benefiting from such setting, each candidate architecture in our search space defines a dynamic sparse sharing route which is more flexible compared with full-sharing in previous approaches. We show that existing typical sharing approaches are sub-graphs in our search space. Extensive experiments on three real-world recommendation datasets demonstrate MTANS achieves consistent improvement compared with single-task models and typical multi-task methods while maintaining high computation efficiency. Furthermore, in-depth experiments demonstrates that MTNAS can learn suitable sparse route to mitigate negative transfer.

Related papers

• Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond [62.406687088097605]
  Multi-Task Learning (MTL) is a framework, where multiple related tasks are learned jointly and benefit from a shared representation space. We show that MTL can be successful with classification tasks with little, or non-overlapping annotations. We propose a novel approach, where knowledge exchange is enabled between the tasks via distribution matching.
  arXiv  Detail & Related papers  (2024-01-02T14:18:11Z)
• Multi-Task Cooperative Learning via Searching for Flat Minima [8.835287696319641]
  We propose to formulate MTL as a multi/bi-level optimization problem, and therefore force features to learn from each task in a cooperative approach. Specifically, we update the sub-model for each task alternatively taking advantage of the learned sub-models of the other tasks. To alleviate the negative transfer problem during the optimization, we search for flat minima for the current objective function.
  arXiv  Detail & Related papers  (2023-09-21T14:00:11Z)
• HiNet: Novel Multi-Scenario & Multi-Task Learning with Hierarchical Information Extraction [50.40732146978222]
  Multi-scenario & multi-task learning has been widely applied to many recommendation systems in industrial applications. We propose a Hierarchical information extraction Network (HiNet) for multi-scenario and multi-task recommendation. HiNet achieves a new state-of-the-art performance and significantly outperforms existing solutions.
  arXiv  Detail & Related papers  (2023-03-10T17:24:41Z)
• Provable Pathways: Learning Multiple Tasks over Multiple Paths [31.43753806123382]
  We develop novel generalization bounds for empirical risk minimization problems learning multiple tasks over multiple paths. In conjunction, we formalize the benefits of resulting multipath representation when adapting to new downstream tasks.
  arXiv  Detail & Related papers  (2023-03-08T02:25:28Z)
• Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners [67.5865966762559]
  We study whether sparsely activated Mixture-of-Experts (MoE) improve multi-task learning. We devise task-aware gating functions to route examples from different tasks to specialized experts. This results in a sparsely activated multi-task model with a large number of parameters, but with the same computational cost as that of a dense model.
  arXiv  Detail & Related papers  (2022-04-16T00:56:12Z)
• Multi-Task Learning as a Bargaining Game [63.49888996291245]
  In Multi-task learning (MTL), a joint model is trained to simultaneously make predictions for several tasks. Since the gradients of these different tasks may conflict, training a joint model for MTL often yields lower performance than its corresponding single-task counterparts. We propose viewing the gradients combination step as a bargaining game, where tasks negotiate to reach an agreement on a joint direction of parameter update.
  arXiv  Detail & Related papers  (2022-02-02T13:21:53Z)
• Semi-supervised Multi-task Learning for Semantics and Depth [88.77716991603252]
  Multi-Task Learning (MTL) aims to enhance the model generalization by sharing representations between related tasks for better performance. We propose the Semi-supervised Multi-Task Learning (MTL) method to leverage the available supervisory signals from different datasets. We present a domain-aware discriminator structure with various alignment formulations to mitigate the domain discrepancy issue among datasets.
  arXiv  Detail & Related papers  (2021-10-14T07:43:39Z)
• Rethinking Hard-Parameter Sharing in Multi-Task Learning [20.792654758645302]
  Hard parameter sharing in multi-task learning (MTL) allows tasks to share some of model parameters, reducing storage cost and improving prediction accuracy. The common sharing practice is to share bottom layers of a deep neural network among tasks while using separate top layers for each task. Using separate bottom-layer parameters could achieve significantly better performance than the common practice.
  arXiv  Detail & Related papers  (2021-07-23T17:26:40Z)
• Learning Twofold Heterogeneous Multi-Task by Sharing Similar Convolution Kernel Pairs [24.044458897098913]
  Heterogeneous multi-task learning (HMTL) is an important topic in multi-task learning (MTL) We design a simple and effective multi-task adaptive learning (MTAL) network to learn multiple tasks in such THMTL setting. Our model effectively performs cross-task learning while suppresses the intra-redundancy of the entire network.
  arXiv  Detail & Related papers  (2021-01-29T06:52:19Z)
• Multi-Task Learning with Deep Neural Networks: A Survey [0.0]
  Multi-task learning (MTL) is a subfield of machine learning in which multiple tasks are simultaneously learned by a shared model. We give an overview of multi-task learning methods for deep neural networks, with the aim of summarizing both the well-established and most recent directions within the field.
  arXiv  Detail & Related papers  (2020-09-10T19:31:04Z)
• MTL-NAS: Task-Agnostic Neural Architecture Search towards General-Purpose Multi-Task Learning [71.90902837008278]
  We propose to incorporate neural architecture search (NAS) into general-purpose multi-task learning (GP-MTL) In order to adapt to different task combinations, we disentangle the GP-MTL networks into single-task backbones. We also propose a novel single-shot gradient-based search algorithm that closes the performance gap between the searched architectures.
  arXiv  Detail & Related papers  (2020-03-31T09:49:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.