Heterogeneous Multi-task Learning with Expert Diversity

• URL: http://arxiv.org/abs/2106.10595v1
• Date: Sun, 20 Jun 2021 01:30:37 GMT
• Title: Heterogeneous Multi-task Learning with Expert Diversity
• Authors: Raquel Aoki, Frederick Tung and Gabriel L. Oliveira
• Abstract summary: We introduce an approach to induce more diversity among experts, thus creating representations more suitable for highly imbalanced and heterogenous MTL learning. We validate our method on three MTL benchmark datasets, including Medical Information Mart for Intensive Care (MIMIC-III) and PubChem BioAssay (PCBA)
• Score: 15.714385295889944
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Predicting multiple heterogeneous biological and medical targets is a challenge for traditional deep learning models. In contrast to single-task learning, in which a separate model is trained for each target, multi-task learning (MTL) optimizes a single model to predict multiple related targets simultaneously. To address this challenge, we propose the Multi-gate Mixture-of-Experts with Exclusivity (MMoEEx). Our work aims to tackle the heterogeneous MTL setting, in which the same model optimizes multiple tasks with different characteristics. Such a scenario can overwhelm current MTL approaches due to the challenges in balancing shared and task-specific representations and the need to optimize tasks with competing optimization paths. Our method makes two key contributions: first, we introduce an approach to induce more diversity among experts, thus creating representations more suitable for highly imbalanced and heterogenous MTL learning; second, we adopt a two-step optimization [6, 11] approach to balancing the tasks at the gradient level. We validate our method on three MTL benchmark datasets, including Medical Information Mart for Intensive Care (MIMIC-III) and PubChem BioAssay (PCBA).

Related papers

• AdapMTL: Adaptive Pruning Framework for Multitask Learning Model [5.643658120200373]
  AdapMTL is an adaptive pruning framework for multitask models. It balances sparsity allocation and accuracy performance across multiple tasks. It showcases superior performance compared to state-of-the-art pruning methods.
  arXiv  Detail & Related papers  (2024-08-07T17:19:15Z)
• MetaGPT: Merging Large Language Models Using Model Exclusive Task Arithmetic [6.46176287368784]
  We propose textbfModel textbfExclusive textbfTask textbfArithmetic for merging textbfGPT-scale models. Our proposed MetaGPT is data-agnostic and bypasses the heavy search process, making it cost-effective and easy to implement for LLMs.
  arXiv  Detail & Related papers  (2024-06-17T10:12:45Z)
• Interpetable Target-Feature Aggregation for Multi-Task Learning based on Bias-Variance Analysis [53.38518232934096]
  Multi-task learning (MTL) is a powerful machine learning paradigm designed to leverage shared knowledge across tasks to improve generalization and performance. We propose an MTL approach at the intersection between task clustering and feature transformation based on a two-phase iterative aggregation of targets and features. In both phases, a key aspect is to preserve the interpretability of the reduced targets and features through the aggregation with the mean, which is motivated by applications to Earth science.
  arXiv  Detail & Related papers  (2024-06-12T08:30:16Z)
• MTLComb: multi-task learning combining regression and classification tasks for joint feature selection [3.708475728683911]
  Multi-task learning (MTL) is a learning paradigm that enables the simultaneous training of multiple communicating algorithms. We propose a provable loss weighting scheme that analytically determines the optimal weights for balancing regression and classification tasks. We introduce MTLComb, an MTL algorithm and software package encompassing optimization procedures, training protocols, and hyper parameter estimation procedures.
  arXiv  Detail & Related papers  (2024-05-16T08:07:25Z)
• Intuition-aware Mixture-of-Rank-1-Experts for Parameter Efficient Finetuning [50.73666458313015]
  Large Language Models (LLMs) have demonstrated significant potential in performing multiple tasks in multimedia applications. MoE has been emerged as a promising solution with its sparse architecture for effective task decoupling. Intuition-MoR1E achieves superior efficiency and 2.15% overall accuracy improvement across 14 public datasets.
  arXiv  Detail & Related papers  (2024-04-13T12:14:58Z)
• When Multi-Task Learning Meets Partial Supervision: A Computer Vision Review [7.776434991976473]
  Multi-Task Learning (MTL) aims to learn multiple tasks simultaneously while exploiting their mutual relationships. This review focuses on how MTL could be utilised under different partial supervision settings to address these challenges.
  arXiv  Detail & Related papers  (2023-07-25T20:08:41Z)
• Multi-task Highly Adaptive Lasso [1.4680035572775534]
  We propose a novel, fully nonparametric approach for the multi-task learning, the Multi-task Highly Adaptive Lasso (MT-HAL) MT-HAL simultaneously learns features, samples and task associations important for the common model, while imposing a shared sparse structure among similar tasks. We show that MT-HAL outperforms sparsity-based MTL competitors across a wide range of simulation studies.
  arXiv  Detail & Related papers  (2023-01-27T23:46:57Z)
• Mod-Squad: Designing Mixture of Experts As Modular Multi-Task Learners [74.92558307689265]
  We propose Mod-Squad, a new model that is Modularized into groups of experts (a 'Squad') We optimize this matching process during the training of a single model. Experiments on the Taskonomy dataset with 13 vision tasks and the PASCAL-Context dataset with 5 vision tasks show the superiority of our approach.
  arXiv  Detail & Related papers  (2022-12-15T18:59:52Z)
• Multi-Task Learning as a Bargaining Game [63.49888996291245]
  In Multi-task learning (MTL), a joint model is trained to simultaneously make predictions for several tasks. Since the gradients of these different tasks may conflict, training a joint model for MTL often yields lower performance than its corresponding single-task counterparts. We propose viewing the gradients combination step as a bargaining game, where tasks negotiate to reach an agreement on a joint direction of parameter update.
  arXiv  Detail & Related papers  (2022-02-02T13:21:53Z)
• Variational Multi-Task Learning with Gumbel-Softmax Priors [105.22406384964144]
  Multi-task learning aims to explore task relatedness to improve individual tasks. We propose variational multi-task learning (VMTL), a general probabilistic inference framework for learning multiple related tasks.
  arXiv  Detail & Related papers  (2021-11-09T18:49:45Z)
• Semi-supervised Multi-task Learning for Semantics and Depth [88.77716991603252]
  Multi-Task Learning (MTL) aims to enhance the model generalization by sharing representations between related tasks for better performance. We propose the Semi-supervised Multi-Task Learning (MTL) method to leverage the available supervisory signals from different datasets. We present a domain-aware discriminator structure with various alignment formulations to mitigate the domain discrepancy issue among datasets.
  arXiv  Detail & Related papers  (2021-10-14T07:43:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.