Attentive Task Interaction Network for Multi-Task Learning

• URL: http://arxiv.org/abs/2201.10649v1
• Date: Tue, 25 Jan 2022 22:03:20 GMT
• Title: Attentive Task Interaction Network for Multi-Task Learning
• Authors: Dimitrios Sinodinos, Narges Armanfard
• Abstract summary: ATI-Net employs knowledge distillation of the latent features for each task, then combines the feature maps to provide improved contextualized information to the decoder. This novel approach to introducing knowledge distillation into an attention based multitask network outperforms state of the art MTL baselines.
• Score: 4.1372815372396525
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multitask learning (MTL) has recently gained a lot of popularity as a learning paradigm that can lead to improved per-task performance while also using fewer per-task model parameters compared to single task learning. One of the biggest challenges regarding MTL networks involves how to share features across tasks. To address this challenge, we propose the Attentive Task Interaction Network (ATI-Net). ATI-Net employs knowledge distillation of the latent features for each task, then combines the feature maps to provide improved contextualized information to the decoder. This novel approach to introducing knowledge distillation into an attention based multitask network outperforms state of the art MTL baselines such as the standalone MTAN and PAD-Net, with roughly the same number of model parameters.

Related papers

• Cross-Task Affinity Learning for Multitask Dense Scene Predictions [5.939164722752263]
  Multitask learning (MTL) has become prominent for its ability to predict multiple tasks jointly. We introduce the Cross-Task Affinity Learning (CTAL) module, a lightweight framework that enhances task refinement in multitask networks. Our results demonstrate state-of-the-art MTL performance for both CNN and transformer backbones, using significantly fewer parameters than single-task learning.
  arXiv  Detail & Related papers  (2024-01-20T05:31:47Z)
• Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond [62.406687088097605]
  Multi-Task Learning (MTL) is a framework, where multiple related tasks are learned jointly and benefit from a shared representation space. We show that MTL can be successful with classification tasks with little, or non-overlapping annotations. We propose a novel approach, where knowledge exchange is enabled between the tasks via distribution matching.
  arXiv  Detail & Related papers  (2024-01-02T14:18:11Z)
• Musketeer: Joint Training for Multi-task Vision Language Model with Task Explanation Prompts [75.75548749888029]
  We present a vision-language model whose parameters are jointly trained on all tasks and fully shared among multiple heterogeneous tasks. With a single model, Musketeer achieves results comparable to or better than strong baselines trained on single tasks, almost uniformly across multiple tasks.
  arXiv  Detail & Related papers  (2023-05-11T17:57:49Z)
• Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners [67.5865966762559]
  We study whether sparsely activated Mixture-of-Experts (MoE) improve multi-task learning. We devise task-aware gating functions to route examples from different tasks to specialized experts. This results in a sparsely activated multi-task model with a large number of parameters, but with the same computational cost as that of a dense model.
  arXiv  Detail & Related papers  (2022-04-16T00:56:12Z)
• Medusa: Universal Feature Learning via Attentional Multitasking [65.94499390875046]
  Recent approaches to multi-task learning have focused on modelling connections between tasks at the decoder level. We argue that MTL is a stepping stone towards universal feature learning (UFL), which is the ability to learn generic features that can be applied to new tasks without retraining. We show the effectiveness of Medusa in UFL (+13.18% improvement) while maintaining MTL performance and being 25% more efficient than previous approaches.
  arXiv  Detail & Related papers  (2022-04-12T10:52:28Z)
• Multi-Task Learning with Deep Neural Networks: A Survey [0.0]
  Multi-task learning (MTL) is a subfield of machine learning in which multiple tasks are simultaneously learned by a shared model. We give an overview of multi-task learning methods for deep neural networks, with the aim of summarizing both the well-established and most recent directions within the field.
  arXiv  Detail & Related papers  (2020-09-10T19:31:04Z)
• Reparameterizing Convolutions for Incremental Multi-Task Learning without Task Interference [75.95287293847697]
  Two common challenges in developing multi-task models are often overlooked in literature. First, enabling the model to be inherently incremental, continuously incorporating information from new tasks without forgetting the previously learned ones (incremental learning) Second, eliminating adverse interactions amongst tasks, which has been shown to significantly degrade the single-task performance in a multi-task setup (task interference)
  arXiv  Detail & Related papers  (2020-07-24T14:44:46Z)
• Knowledge Distillation for Multi-task Learning [38.20005345733544]
  Multi-task learning (MTL) is to learn one single model that performs multiple tasks for achieving good performance on all tasks and lower cost on computation. Learning such a model requires to jointly optimize losses of a set of tasks with different difficulty levels, magnitudes, and characteristics. We propose a knowledge distillation based method in this work to address the imbalance problem in multi-task learning.
  arXiv  Detail & Related papers  (2020-07-14T08:02:42Z)
• MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning [82.62433731378455]
  We show that tasks with high affinity at a certain scale are not guaranteed to retain this behaviour at other scales. We propose a novel architecture, namely MTI-Net, that builds upon this finding.
  arXiv  Detail & Related papers  (2020-01-19T21:02:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.