FairBranch: Mitigating Bias Transfer in Fair Multi-task Learning

• URL: http://arxiv.org/abs/2310.13746v2
• Date: Tue, 24 Sep 2024 14:06:33 GMT
• Title: FairBranch: Mitigating Bias Transfer in Fair Multi-task Learning
• Authors: Arjun Roy, Christos Koutlis, Symeon Papadopoulos, Eirini Ntoutsi,
• Abstract summary: Multi-Task Learning (MTL) suffers when unrelated tasks negatively impact each other by updating shared parameters with conflicting gradients. This is known as negative transfer and leads to a drop in MTL accuracy compared to single-task learning (STL)
• Score: 15.319254128769973
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The generalisation capacity of Multi-Task Learning (MTL) suffers when unrelated tasks negatively impact each other by updating shared parameters with conflicting gradients. This is known as negative transfer and leads to a drop in MTL accuracy compared to single-task learning (STL). Lately, there has been a growing focus on the fairness of MTL models, requiring the optimization of both accuracy and fairness for individual tasks. Analogously to negative transfer for accuracy, task-specific fairness considerations might adversely affect the fairness of other tasks when there is a conflict of fairness loss gradients between the jointly learned tasks - we refer to this as Bias Transfer. To address both negative- and bias-transfer in MTL, we propose a novel method called FairBranch, which branches the MTL model by assessing the similarity of learned parameters, thereby grouping related tasks to alleviate negative transfer. Moreover, it incorporates fairness loss gradient conflict correction between adjoining task-group branches to address bias transfer within these task groups. Our experiments on tabular and visual MTL problems show that FairBranch outperforms state-of-the-art MTLs on both fairness and accuracy.

Related papers

• Mitigate Negative Transfer with Similarity Heuristic Lifelong Prompt Tuning [26.949872705635084]
  We present the Similarity Heuristic Lifelong Prompt Tuning (SH) framework. SH partitions tasks into two distinct subsets by harnessing a learnable similarity metric. Our experiments shows that SH outperforms state-of-the-art techniques in lifelong learning benchmarks.
  arXiv  Detail & Related papers  (2024-06-18T03:57:49Z)
• Fair Resource Allocation in Multi-Task Learning [12.776767874217663]
  Multi-task learning (MTL) can leverage the shared knowledge across tasks, resulting in improved data efficiency and generalization performance. A major challenge in MTL lies in the presence of conflicting gradients, which can hinder the fair optimization of some tasks. Inspired by fair resource allocation in communication networks, we propose FairGrad, a novel MTL optimization method.
  arXiv  Detail & Related papers  (2024-02-23T22:46:14Z)
• Multitask Learning Can Improve Worst-Group Outcomes [76.92646345152788]
  Multitask learning (MTL) is one such widely used technique. We propose to modify standard MTL by regularizing the joint multitask representation space. We find that our regularized MTL approach emphconsistently outperforms JTT on both average and worst-group outcomes.
  arXiv  Detail & Related papers  (2023-12-05T21:38:24Z)
• Addressing Negative Transfer in Diffusion Models [25.457422246404853]
  Multi-task learning (MTL) can lead to negative transfer in diffusion models. We propose clustering the denoising tasks into small task clusters and applying MTL methods to them. We show that interval clustering can be solved using dynamic programming, utilizing signal-to-noise ratio, timestep, and task affinity for clustering objectives.
  arXiv  Detail & Related papers  (2023-06-01T05:17:07Z)
• ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning [59.08197876733052]
  Auxiliary-Task Learning (ATL) aims to improve the performance of the target task by leveraging the knowledge obtained from related tasks. Sometimes, learning multiple tasks simultaneously results in lower accuracy than learning only the target task, known as negative transfer. ForkMerge is a novel approach that periodically forks the model into multiple branches, automatically searches the varying task weights.
  arXiv  Detail & Related papers  (2023-01-30T02:27:02Z)
• Learning to Teach Fairness-aware Deep Multi-task Learning [17.30805079904122]
  We propose a flexible approach that learns how to be fair in a multi-task setting by selecting which objective (accuracy or fairness) to optimize at each step. Experiments on three real datasets show that L2T-FMT improves on both fairness (12-19%) and accuracy (up to 2%) over state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-06-16T18:43:16Z)
• FedGradNorm: Personalized Federated Gradient-Normalized Multi-Task Learning [50.756991828015316]
  Multi-task learning (MTL) is a novel framework to learn several tasks simultaneously with a single shared network. We propose FedGradNorm which uses a dynamic-weighting method to normalize norms in order to balance learning speeds among different tasks.
  arXiv  Detail & Related papers  (2022-03-24T17:43:12Z)
• Multi-Task Learning as a Bargaining Game [63.49888996291245]
  In Multi-task learning (MTL), a joint model is trained to simultaneously make predictions for several tasks. Since the gradients of these different tasks may conflict, training a joint model for MTL often yields lower performance than its corresponding single-task counterparts. We propose viewing the gradients combination step as a bargaining game, where tasks negotiate to reach an agreement on a joint direction of parameter update.
  arXiv  Detail & Related papers  (2022-02-02T13:21:53Z)
• Task Uncertainty Loss Reduce Negative Transfer in Asymmetric Multi-task Feature Learning [0.0]
  Multi-task learning (MTL) can improve task performance overall relative to single-task learning (STL), but can hide negative transfer (NT) Asymmetric multitask feature learning (AMTFL) is an approach that tries to address this by allowing tasks with higher loss values to have smaller influence on feature representations for learning other tasks. We present examples of NT in two datasets (image recognition and pharmacogenomics) and tackle this challenge by using aleatoric homoscedastic uncertainty to capture the relative confidence between tasks, and set weights for task loss.
  arXiv  Detail & Related papers  (2020-12-17T13:30:45Z)
• Precise High-Dimensional Asymptotics for Quantifying Heterogeneous Transfers [66.66228496844191]
  The problem of learning one task with samples from another task is central to transfer learning (TL) In this paper, we examine a fundamental question: When does combining the data samples from a source task and a target task perform better than single-task learning with the target task alone?
  arXiv  Detail & Related papers  (2020-10-22T14:14:20Z)
• Towards Accurate Knowledge Transfer via Target-awareness Representation Disentanglement [56.40587594647692]
  We propose a novel transfer learning algorithm, introducing the idea of Target-awareness REpresentation Disentanglement (TRED) TRED disentangles the relevant knowledge with respect to the target task from the original source model and used as a regularizer during fine-tuning the target model. Experiments on various real world datasets show that our method stably improves the standard fine-tuning by more than 2% in average.
  arXiv  Detail & Related papers  (2020-10-16T17:45:08Z)
• Task-Feature Collaborative Learning with Application to Personalized Attribute Prediction [166.87111665908333]
  We propose a novel multi-task learning method called Task-Feature Collaborative Learning (TFCL) Specifically, we first propose a base model with a heterogeneous block-diagonal structure regularizer to leverage the collaborative grouping of features and tasks. As a practical extension, we extend the base model by allowing overlapping features and differentiating the hard tasks.
  arXiv  Detail & Related papers  (2020-04-29T02:32:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.