Safe Multi-Task Learning
- URL: http://arxiv.org/abs/2111.10601v1
- Date: Sat, 20 Nov 2021 14:21:02 GMT
- Title: Safe Multi-Task Learning
- Authors: Pengxin Guo, Feiyang Ye, and Yu Zhang
- Abstract summary: We propose a Safe Multi-Task Learning (SMTL) model, which consists of a public encoder shared by all the tasks, private encoders, gates, and private decoders.
To reduce the storage cost during the inference stage, a lite version of SMTL is proposed to allow the gate to choose either the public encoder or the corresponding private encoder.
- Score: 3.508126539399186
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In recent years, Multi-Task Learning (MTL) attracts much attention due to its
good performance in many applications. However, many existing MTL models cannot
guarantee that its performance is no worse than its single-task counterpart on
each task. Though this phenomenon has been empirically observed by some works,
little work aims to handle the resulting problem, which is formally defined as
negative sharing in this paper. To achieve safe multi-task learning where no
\textit{negative sharing} occurs, we propose a Safe Multi-Task Learning (SMTL)
model, which consists of a public encoder shared by all the tasks, private
encoders, gates, and private decoders. Specifically, each task has a private
encoder, a gate, and a private decoder, where the gate is to learn how to
combine the private encoder and public encoder for the downstream private
decoder. To reduce the storage cost during the inference stage, a lite version
of SMTL is proposed to allow the gate to choose either the public encoder or
the corresponding private encoder. Moreover, we propose a variant of SMTL to
place all the gates after decoders of all the tasks. Experiments on several
benchmark datasets demonstrate the effectiveness of the proposed methods.
Related papers
- MFTCoder: Boosting Code LLMs with Multitask Fine-Tuning [28.12788291168137]
We present a multi-task fine-tuning framework, MFTcoder, that enables simultaneous and parallel fine-tuning on multiple tasks.
Experiments have conclusively demonstrated that our multi-task fine-tuning approach outperforms both individual fine-tuning on single tasks and fine-tuning on a mixed ensemble of tasks.
arXiv Detail & Related papers (2023-11-04T02:22:40Z) - Deformable Mixer Transformer with Gating for Multi-Task Learning of
Dense Prediction [126.34551436845133]
CNNs and Transformers have their own advantages and both have been widely used for dense prediction in multi-task learning (MTL)
We present a novel MTL model by combining both merits of deformable CNN and query-based Transformer with shared gating for multi-task learning of dense prediction.
arXiv Detail & Related papers (2023-08-10T17:37:49Z) - MetaMorphosis: Task-oriented Privacy Cognizant Feature Generation for
Multi-task Learning [6.056197449765416]
This paper proposes a novel deep learning-based privacy-cognizant feature generation process called MetaMorphosis.
We show that MetaMorphosis outperforms recent adversarial learning and universal feature generation methods by guaranteeing privacy requirements.
arXiv Detail & Related papers (2023-05-13T01:59:07Z) - A Study of Autoregressive Decoders for Multi-Tasking in Computer Vision [93.90545426665999]
We take a close look at autoregressive decoders for multi-task learning in multimodal computer vision.
A key finding is that a small decoder learned on top of a frozen pretrained encoder works surprisingly well.
It can be seen as teaching a decoder to interact with a pretrained vision model via natural language.
arXiv Detail & Related papers (2023-03-30T13:42:58Z) - M$^3$ViT: Mixture-of-Experts Vision Transformer for Efficient Multi-task
Learning with Model-Accelerator Co-design [95.41238363769892]
Multi-task learning (MTL) encapsulates multiple learned tasks in a single model and often lets those tasks learn better jointly.
Current MTL regimes have to activate nearly the entire model even to just execute a single task.
We present a model-accelerator co-design framework to enable efficient on-device MTL.
arXiv Detail & Related papers (2022-10-26T15:40:24Z) - String-based Molecule Generation via Multi-decoder VAE [56.465033997245776]
We investigate the problem of string-based molecular generation via variational autoencoders (VAEs)
We propose a simple, yet effective idea to improve the performance of VAE for the task.
In our experiments, the proposed VAE model particularly performs well for generating a sample from out-of-domain distribution.
arXiv Detail & Related papers (2022-08-23T03:56:30Z) - LAVENDER: Unifying Video-Language Understanding as Masked Language
Modeling [102.42424022921243]
Masked Language Modeling (MLM) is used as the common interface for all pre-training and downstream tasks.
Experiments show that this unified framework achieves competitive performance on 14 VidL benchmarks.
arXiv Detail & Related papers (2022-06-14T20:43:25Z) - Joint Multiple Intent Detection and Slot Filling via Self-distillation [29.17761742391222]
Intent detection and slot filling are two main tasks in natural language understanding (NLU) for identifying users' needs from their utterances.
Most previous works assume that each utterance only corresponds to one intent, ignoring the fact that a user utterance in many cases could include multiple intents.
We propose a novel Self-Distillation Joint NLU model (SDJN) for multi-intent NLU.
arXiv Detail & Related papers (2021-08-18T08:45:03Z) - MIST: Multiple Instance Self-Training Framework for Video Anomaly
Detection [76.80153360498797]
We develop a multiple instance self-training framework (MIST) to efficiently refine task-specific discriminative representations.
MIST is composed of 1) a multiple instance pseudo label generator, which adapts a sparse continuous sampling strategy to produce more reliable clip-level pseudo labels, and 2) a self-guided attention boosted feature encoder.
Our method performs comparably to or even better than existing supervised and weakly supervised methods, specifically obtaining a frame-level AUC 94.83% on ShanghaiTech.
arXiv Detail & Related papers (2021-04-04T15:47:14Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.