Variational Multi-Task Learning with Gumbel-Softmax Priors

• URL: http://arxiv.org/abs/2111.05323v1
• Date: Tue, 9 Nov 2021 18:49:45 GMT
• Title: Variational Multi-Task Learning with Gumbel-Softmax Priors
• Authors: Jiayi Shen, Xiantong Zhen, Marcel Worring, Ling Shao
• Abstract summary: Multi-task learning aims to explore task relatedness to improve individual tasks. We propose variational multi-task learning (VMTL), a general probabilistic inference framework for learning multiple related tasks.
• Score: 105.22406384964144
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-task learning aims to explore task relatedness to improve individual tasks, which is of particular significance in the challenging scenario that only limited data is available for each task. To tackle this challenge, we propose variational multi-task learning (VMTL), a general probabilistic inference framework for learning multiple related tasks. We cast multi-task learning as a variational Bayesian inference problem, in which task relatedness is explored in a unified manner by specifying priors. To incorporate shared knowledge into each task, we design the prior of a task to be a learnable mixture of the variational posteriors of other related tasks, which is learned by the Gumbel-Softmax technique. In contrast to previous methods, our VMTL can exploit task relatedness for both representations and classifiers in a principled way by jointly inferring their posteriors. This enables individual tasks to fully leverage inductive biases provided by related tasks, therefore improving the overall performance of all tasks. Experimental results demonstrate that the proposed VMTL is able to effectively tackle a variety of challenging multi-task learning settings with limited training data for both classification and regression. Our method consistently surpasses previous methods, including strong Bayesian approaches, and achieves state-of-the-art performance on five benchmark datasets.

Related papers

• Identifying Task Groupings for Multi-Task Learning Using Pointwise V-Usable Information [11.545160697026514]
  We propose a metric of task relatedness based on task difficulty measured by pointwise V-usable information (PVI) We conduct experiments to evaluate the feasibility of this metric for task grouping on 15 NLP datasets in the general, biomedical, and clinical domains.
  arXiv  Detail & Related papers  (2024-10-16T17:49:45Z)
• CoTBal: Comprehensive Task Balancing for Multi-Task Visual Instruction Tuning [20.58878416527427]
  We propose a novel Comprehensive Task Balancing algorithm for multi-task visual instruction tuning of LMMs. Our CoTBal leads to superior overall performance in multi-task visual instruction tuning.
  arXiv  Detail & Related papers  (2024-03-07T09:11:16Z)
• Data-CUBE: Data Curriculum for Instruction-based Sentence Representation Learning [85.66907881270785]
  We propose a data curriculum method, namely Data-CUBE, that arranges the orders of all the multi-task data for training. In the task level, we aim to find the optimal task order to minimize the total cross-task interference risk. In the instance level, we measure the difficulty of all instances per task, then divide them into the easy-to-difficult mini-batches for training.
  arXiv  Detail & Related papers  (2024-01-07T18:12:20Z)
• Task Selection and Assignment for Multi-modal Multi-task Dialogue Act Classification with Non-stationary Multi-armed Bandits [11.682678945754837]
  Multi-task learning (MTL) aims to improve the performance of a primary task by jointly learning with related auxiliary tasks. Previous studies suggest that such a random selection of tasks may not be helpful, and can even be harmful to performance. This paper proposes a method for selecting and assigning tasks based on non-stationary multi-armed bandits.
  arXiv  Detail & Related papers  (2023-09-18T14:51:51Z)
• Multi-task Bias-Variance Trade-off Through Functional Constraints [102.64082402388192]
  Multi-task learning aims to acquire a set of functions that perform well for diverse tasks. In this paper we draw intuition from the two extreme learning scenarios -- a single function for all tasks, and a task-specific function that ignores the other tasks. We introduce a constrained learning formulation that enforces domain specific solutions to a central function.
  arXiv  Detail & Related papers  (2022-10-27T16:06:47Z)
• Leveraging convergence behavior to balance conflicting tasks in multi-task learning [3.6212652499950138]
  Multi-Task Learning uses correlated tasks to improve performance generalization. Tasks often conflict with each other, which makes it challenging to define how the gradients of multiple tasks should be combined. We propose a method that takes into account temporal behaviour of the gradients to create a dynamic bias that adjust the importance of each task during the backpropagation.
  arXiv  Detail & Related papers  (2022-04-14T01:52:34Z)
• On Steering Multi-Annotations per Sample for Multi-Task Learning [79.98259057711044]
  The study of multi-task learning has drawn great attention from the community. Despite the remarkable progress, the challenge of optimally learning different tasks simultaneously remains to be explored. Previous works attempt to modify the gradients from different tasks. Yet these methods give a subjective assumption of the relationship between tasks, and the modified gradient may be less accurate. In this paper, we introduce Task Allocation(STA), a mechanism that addresses this issue by a task allocation approach, in which each sample is randomly allocated a subset of tasks. For further progress, we propose Interleaved Task Allocation(ISTA) to iteratively allocate all
  arXiv  Detail & Related papers  (2022-03-06T11:57:18Z)
• Semi-supervised Multi-task Learning for Semantics and Depth [88.77716991603252]
  Multi-Task Learning (MTL) aims to enhance the model generalization by sharing representations between related tasks for better performance. We propose the Semi-supervised Multi-Task Learning (MTL) method to leverage the available supervisory signals from different datasets. We present a domain-aware discriminator structure with various alignment formulations to mitigate the domain discrepancy issue among datasets.
  arXiv  Detail & Related papers  (2021-10-14T07:43:39Z)
• Measuring and Harnessing Transference in Multi-Task Learning [58.48659733262734]
  Multi-task learning can leverage information learned by one task to benefit the training of other tasks. We analyze the dynamics of information transfer, or transference, across tasks throughout training.
  arXiv  Detail & Related papers  (2020-10-29T08:25:43Z)
• A Simple General Approach to Balance Task Difficulty in Multi-Task Learning [4.531240717484252]
  In multi-task learning, difficulty levels of different tasks are varying. We propose a Balanced Multi-Task Learning (BMTL) framework. The proposed BMTL framework is very simple and it can be combined with most multi-task learning models.
  arXiv  Detail & Related papers  (2020-02-12T04:31:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.