Robust Multi-Task Learning with Excess Risks

• URL: http://arxiv.org/abs/2402.02009v3
• Date: Thu, 18 Jul 2024 18:43:26 GMT
• Title: Robust Multi-Task Learning with Excess Risks
• Authors: Yifei He, Shiji Zhou, Guojun Zhang, Hyokun Yun, Yi Xu, Belinda Zeng, Trishul Chilimbi, Han Zhao,
• Abstract summary: Multi-task learning (MTL) considers learning a joint model for multiple tasks by optimizing a convex combination of all task losses. Existing methods use an adaptive weight updating scheme, where task weights are dynamically adjusted based on their respective losses to prioritize difficult tasks. We propose Multi-Task Learning with Excess Risks (ExcessMTL), an excess risk-based task balancing method that updates the task weights by their distances to convergence.
• Score: 24.695243608197835
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-task learning (MTL) considers learning a joint model for multiple tasks by optimizing a convex combination of all task losses. To solve the optimization problem, existing methods use an adaptive weight updating scheme, where task weights are dynamically adjusted based on their respective losses to prioritize difficult tasks. However, these algorithms face a great challenge whenever label noise is present, in which case excessive weights tend to be assigned to noisy tasks that have relatively large Bayes optimal errors, thereby overshadowing other tasks and causing performance to drop across the board. To overcome this limitation, we propose Multi-Task Learning with Excess Risks (ExcessMTL), an excess risk-based task balancing method that updates the task weights by their distances to convergence instead. Intuitively, ExcessMTL assigns higher weights to worse-trained tasks that are further from convergence. To estimate the excess risks, we develop an efficient and accurate method with Taylor approximation. Theoretically, we show that our proposed algorithm achieves convergence guarantees and Pareto stationarity. Empirically, we evaluate our algorithm on various MTL benchmarks and demonstrate its superior performance over existing methods in the presence of label noise. Our code is available at https://github.com/yifei-he/ExcessMTL.

Related papers

• Analytical Uncertainty-Based Loss Weighting in Multi-Task Learning [8.493889694402478]
  Key challenge in multi-task learning (MTL) is balancing individual task losses during neural network training to improve performance and efficiency. We propose a novel task-weighting method by building on the most prevalent approach of Uncertainty Weighting. Our approach yields comparable results to the analyticallyly prohibitive, brute-force approach of Scalarization.
  arXiv  Detail & Related papers  (2024-08-15T07:10:17Z)
• Data-CUBE: Data Curriculum for Instruction-based Sentence Representation Learning [85.66907881270785]
  We propose a data curriculum method, namely Data-CUBE, that arranges the orders of all the multi-task data for training. In the task level, we aim to find the optimal task order to minimize the total cross-task interference risk. In the instance level, we measure the difficulty of all instances per task, then divide them into the easy-to-difficult mini-batches for training.
  arXiv  Detail & Related papers  (2024-01-07T18:12:20Z)
• Low-Rank Multitask Learning based on Tensorized SVMs and LSSVMs [65.42104819071444]
  Multitask learning (MTL) leverages task-relatedness to enhance performance. We employ high-order tensors, with each mode corresponding to a task index, to naturally represent tasks referenced by multiple indices. We propose a general framework of low-rank MTL methods with tensorized support vector machines (SVMs) and least square support vector machines (LSSVMs)
  arXiv  Detail & Related papers  (2023-08-30T14:28:26Z)
• Multi-Objective Optimization for Sparse Deep Multi-Task Learning [0.0]
  We present a Multi-Objective Optimization algorithm using a modified Weighted Chebyshev scalarization for training Deep Neural Networks (DNNs) Our work aims to address the (economical and also ecological) sustainability issue of DNN models, with particular focus on Deep Multi-Task models.
  arXiv  Detail & Related papers  (2023-08-23T16:42:27Z)
• FAMO: Fast Adaptive Multitask Optimization [48.59232177073481]
  We introduce Fast Adaptive Multitask Optimization FAMO, a dynamic weighting method that decreases task losses in a balanced way. Our results indicate that FAMO achieves comparable or superior performance to state-of-the-art gradient manipulation techniques.
  arXiv  Detail & Related papers  (2023-06-06T15:39:54Z)
• New Tight Relaxations of Rank Minimization for Multi-Task Learning [161.23314844751556]
  We propose two novel multi-task learning formulations based on two regularization terms. We show that our methods can correctly recover the low-rank structure shared across tasks, and outperform related multi-task learning methods.
  arXiv  Detail & Related papers  (2021-12-09T07:29:57Z)
• Conflict-Averse Gradient Descent for Multi-task Learning [56.379937772617]
  A major challenge in optimizing a multi-task model is the conflicting gradients. We introduce Conflict-Averse Gradient descent (CAGrad) which minimizes the average loss function. CAGrad balances the objectives automatically and still provably converges to a minimum over the average loss.
  arXiv  Detail & Related papers  (2021-10-26T22:03:51Z)
• Multi-Task Meta-Learning Modification with Stochastic Approximation [0.7734726150561089]
  A few-shot learning problem is one of the main benchmarks of meta-learning algorithms. In this paper we investigate the modification of standard meta-learning pipeline that takes a multi-task approach during training. The proposed method simultaneously utilizes information from several meta-training tasks in a common loss function. Proper optimization of these weights can have a big influence on training of the entire model and might improve the quality on test time tasks.
  arXiv  Detail & Related papers  (2021-10-25T18:11:49Z)
• SLAW: Scaled Loss Approximate Weighting for Efficient Multi-Task Learning [0.0]
  Multi-task learning (MTL) is a subfield of machine learning with important applications. The best MTL optimization methods require individually computing the gradient of each task's loss function. We propose Scaled Loss Approximate Weighting (SLAW), a method for multi-task optimization that matches the performance of the best existing methods while being much more efficient.
  arXiv  Detail & Related papers  (2021-09-16T20:58:40Z)
• Dynamic Multi-Robot Task Allocation under Uncertainty and Temporal Constraints [52.58352707495122]
  We present a multi-robot allocation algorithm that decouples the key computational challenges of sequential decision-making under uncertainty and multi-agent coordination. We validate our results over a wide range of simulations on two distinct domains: multi-arm conveyor belt pick-and-place and multi-drone delivery dispatch in a city.
  arXiv  Detail & Related papers  (2020-05-27T01:10:41Z)
• A Simple General Approach to Balance Task Difficulty in Multi-Task Learning [4.531240717484252]
  In multi-task learning, difficulty levels of different tasks are varying. We propose a Balanced Multi-Task Learning (BMTL) framework. The proposed BMTL framework is very simple and it can be combined with most multi-task learning models.
  arXiv  Detail & Related papers  (2020-02-12T04:31:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.