Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspective

• URL: http://arxiv.org/abs/2211.13723v5
• Date: Mon, 29 Sep 2025 14:14:15 GMT
• Title: Beyond Losses Reweighting: Empowering Multi-Task Learning via the Generalization Perspective
• Authors: Hoang Phan, Lam Tran, Quyen Tran, Ngoc N. Tran, Tuan Truong, Qi Lei, Nhat Ho, Dinh Phung, Trung Le,
• Abstract summary: Multi-task learning (MTL) trains deep neural networks to optimize several objectives simultaneously using a shared backbone.<n>We introduce a novel MTL framework that leverages weight perturbation to regulate gradient norms, thus improving generalization.<n>Our method significantly outperforms existing gradient-based MTL techniques in terms of task performance and overall model robustness.
• Score: 61.10883077161432
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Multi-task learning (MTL) trains deep neural networks to optimize several objectives simultaneously using a shared backbone, which leads to reduced computational costs, improved data efficiency, and enhanced performance through cross-task knowledge sharing. Although recent gradient manipulation techniques aim to find a common descent direction that benefits all tasks, conventional empirical loss minimization still leaves models vulnerable to overfitting and gradient conflicts. To address this, we introduce a novel MTL framework that leverages weight perturbation to regulate gradient norms, thus improving generalization. By adaptively modulating weight perturbations, our approach harmonizes task-specific gradients, reducing conflicts and encouraging more robust learning across tasks. Theoretical insights reveal that controlling the gradient norm through weight perturbation directly contributes to better generalization. Extensive experiments across diverse applications demonstrate that our method significantly outperforms existing gradient-based MTL techniques in terms of task performance and overall model robustness.

Related papers

• Revisit the Imbalance Optimization in Multi-task Learning: An Experimental Analysis [44.410446932443]
  Multi-task learning (MTL) aims to build general-purpose vision systems by training a single network to perform multiple tasks jointly.<n>While promising, its potential is often hindered by "unbalanced optimization"<n>This paper presents a systematic experimental analysis to dissect the factors contributing to this persistent problem.
  arXiv  Detail & Related papers  (2025-09-28T14:40:06Z)
• Efficient Domain Adaptation of Multimodal Embeddings using Constrastive Learning [0.08192907805418582]
  Current approaches either yield subpar results when using pretrained models without task-specific adaptation, or require substantial computational resources for fine-tuning. We propose a novel method for adapting foundational, multimodal embeddings to downstream tasks, without the need of expensive fine-tuning processes.
  arXiv  Detail & Related papers  (2025-02-04T06:30:12Z)
• Fantastic Multi-Task Gradient Updates and How to Find Them In a Cone [14.702480423653984]
  We propose ConicGrad, a principled, scalable, and robust MTL approach formulated as a constrained optimization problem.<n>Our method introduces an angular constraint to dynamically regulate gradient update directions, confining them within a cone centered on the reference gradient of the overall objective.<n>We conduct extensive experiments on standard supervised learning and reinforcement learning MTL benchmarks, and demonstrate that ConicGrad achieves state-of-the-art performance across diverse tasks.
  arXiv  Detail & Related papers  (2025-01-31T23:11:12Z)
• Unlearning as multi-task optimization: A normalized gradient difference approach with an adaptive learning rate [105.86576388991713]
  We introduce a normalized gradient difference (NGDiff) algorithm, enabling us to have better control over the trade-off between the objectives.<n>We provide a theoretical analysis and empirically demonstrate the superior performance of NGDiff among state-of-the-art unlearning methods on the TOFU and MUSE datasets.
  arXiv  Detail & Related papers  (2024-10-29T14:41:44Z)
• Fair Resource Allocation in Multi-Task Learning [12.776767874217663]
  Multi-task learning (MTL) can leverage the shared knowledge across tasks, resulting in improved data efficiency and generalization performance. A major challenge in MTL lies in the presence of conflicting gradients, which can hinder the fair optimization of some tasks. Inspired by fair resource allocation in communication networks, we propose FairGrad, a novel MTL optimization method.
  arXiv  Detail & Related papers  (2024-02-23T22:46:14Z)
• Towards Principled Task Grouping for Multi-Task Learning [12.757893623250252]
  Multi-task learning (MTL) aims to leverage shared information among tasks to improve learning efficiency and accuracy.<n> MTL often struggles to effectively manage positive and negative transfer between tasks, which can hinder performance improvements.<n>This paper introduces a principled approach to task grouping in MTL, advancing beyond existing methods by addressing key theoretical and practical limitations.
  arXiv  Detail & Related papers  (2024-02-23T13:51:20Z)
• Robust Analysis of Multi-Task Learning Efficiency: New Benchmarks on Light-Weighed Backbones and Effective Measurement of Multi-Task Learning Challenges by Feature Disentanglement [69.51496713076253]
  In this paper, we focus on the aforementioned efficiency aspects of existing MTL methods. We first carry out large-scale experiments of the methods with smaller backbones and on a the MetaGraspNet dataset as a new test ground. We also propose Feature Disentanglement measure as a novel and efficient identifier of the challenges in MTL.
  arXiv  Detail & Related papers  (2024-02-05T22:15:55Z)
• Data-CUBE: Data Curriculum for Instruction-based Sentence Representation Learning [85.66907881270785]
  We propose a data curriculum method, namely Data-CUBE, that arranges the orders of all the multi-task data for training. In the task level, we aim to find the optimal task order to minimize the total cross-task interference risk. In the instance level, we measure the difficulty of all instances per task, then divide them into the easy-to-difficult mini-batches for training.
  arXiv  Detail & Related papers  (2024-01-07T18:12:20Z)
• Multi-Task Cooperative Learning via Searching for Flat Minima [8.835287696319641]
  We propose to formulate MTL as a multi/bi-level optimization problem, and therefore force features to learn from each task in a cooperative approach. Specifically, we update the sub-model for each task alternatively taking advantage of the learned sub-models of the other tasks. To alleviate the negative transfer problem during the optimization, we search for flat minima for the current objective function.
  arXiv  Detail & Related papers  (2023-09-21T14:00:11Z)
• Equitable Multi-task Learning [18.65048321820911]
  Multi-task learning (MTL) has achieved great success in various research domains, such as CV, NLP and IR. We propose a novel multi-task optimization method, named EMTL, to achieve equitable MTL. Our method stably outperforms state-of-the-art methods on the public benchmark datasets of two different research domains.
  arXiv  Detail & Related papers  (2023-06-15T03:37:23Z)
• Multi-Task Learning as a Bargaining Game [63.49888996291245]
  In Multi-task learning (MTL), a joint model is trained to simultaneously make predictions for several tasks. Since the gradients of these different tasks may conflict, training a joint model for MTL often yields lower performance than its corresponding single-task counterparts. We propose viewing the gradients combination step as a bargaining game, where tasks negotiate to reach an agreement on a joint direction of parameter update.
  arXiv  Detail & Related papers  (2022-02-02T13:21:53Z)
• Conflict-Averse Gradient Descent for Multi-task Learning [56.379937772617]
  A major challenge in optimizing a multi-task model is the conflicting gradients. We introduce Conflict-Averse Gradient descent (CAGrad) which minimizes the average loss function. CAGrad balances the objectives automatically and still provably converges to a minimum over the average loss.
  arXiv  Detail & Related papers  (2021-10-26T22:03:51Z)
• SLAW: Scaled Loss Approximate Weighting for Efficient Multi-Task Learning [0.0]
  Multi-task learning (MTL) is a subfield of machine learning with important applications. The best MTL optimization methods require individually computing the gradient of each task's loss function. We propose Scaled Loss Approximate Weighting (SLAW), a method for multi-task optimization that matches the performance of the best existing methods while being much more efficient.
  arXiv  Detail & Related papers  (2021-09-16T20:58:40Z)
• Task-Feature Collaborative Learning with Application to Personalized Attribute Prediction [166.87111665908333]
  We propose a novel multi-task learning method called Task-Feature Collaborative Learning (TFCL) Specifically, we first propose a base model with a heterogeneous block-diagonal structure regularizer to leverage the collaborative grouping of features and tasks. As a practical extension, we extend the base model by allowing overlapping features and differentiating the hard tasks.
  arXiv  Detail & Related papers  (2020-04-29T02:32:04Z)
• Regularizing Meta-Learning via Gradient Dropout [102.29924160341572]
  meta-learning models are prone to overfitting when there are no sufficient training tasks for the meta-learners to generalize. We introduce a simple yet effective method to alleviate the risk of overfitting for gradient-based meta-learning.
  arXiv  Detail & Related papers  (2020-04-13T10:47:02Z)
• A Simple General Approach to Balance Task Difficulty in Multi-Task Learning [4.531240717484252]
  In multi-task learning, difficulty levels of different tasks are varying. We propose a Balanced Multi-Task Learning (BMTL) framework. The proposed BMTL framework is very simple and it can be combined with most multi-task learning models.
  arXiv  Detail & Related papers  (2020-02-12T04:31:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.