Gap Minimization for Knowledge Sharing and Transfer
- URL: http://arxiv.org/abs/2201.11231v1
- Date: Wed, 26 Jan 2022 23:06:20 GMT
- Title: Gap Minimization for Knowledge Sharing and Transfer
- Authors: Boyu Wang, Jorge Mendez, Changjian Shui, Fan Zhou, Di Wu, Christian
Gagn\'e, Eric Eaton
- Abstract summary: In this paper, we introduce the notion of emphperformance gap, an intuitive and novel measure of the distance between learning tasks.
We show that the performance gap can be viewed as a data- and algorithm-dependent regularizer, which controls the model complexity and leads to finer guarantees.
We instantiate this principle with two algorithms: 1. gapBoost, a novel and principled boosting algorithm that explicitly minimizes the performance gap between source and target domains for transfer learning; and 2. gapMTNN, a representation learning algorithm that reformulates gap minimization as semantic conditional matching
- Score: 24.954256258648982
- License: http://creativecommons.org/publicdomain/zero/1.0/
- Abstract: Learning from multiple related tasks by knowledge sharing and transfer has
become increasingly relevant over the last two decades. In order to
successfully transfer information from one task to another, it is critical to
understand the similarities and differences between the domains. In this paper,
we introduce the notion of \emph{performance gap}, an intuitive and novel
measure of the distance between learning tasks. Unlike existing measures which
are used as tools to bound the difference of expected risks between tasks
(e.g., $\mathcal{H}$-divergence or discrepancy distance), we theoretically show
that the performance gap can be viewed as a data- and algorithm-dependent
regularizer, which controls the model complexity and leads to finer guarantees.
More importantly, it also provides new insights and motivates a novel principle
for designing strategies for knowledge sharing and transfer: gap minimization.
We instantiate this principle with two algorithms: 1. {gapBoost}, a novel and
principled boosting algorithm that explicitly minimizes the performance gap
between source and target domains for transfer learning; and 2. {gapMTNN}, a
representation learning algorithm that reformulates gap minimization as
semantic conditional matching for multitask learning. Our extensive evaluation
on both transfer learning and multitask learning benchmark data sets shows that
our methods outperform existing baselines.
Related papers
- Multitask Learning with No Regret: from Improved Confidence Bounds to
Active Learning [79.07658065326592]
Quantifying uncertainty in the estimated tasks is of pivotal importance for many downstream applications, such as online or active learning.
We provide novel multitask confidence intervals in the challenging setting when neither the similarity between tasks nor the tasks' features are available to the learner.
We propose a novel online learning algorithm that achieves such improved regret without knowing this parameter in advance.
arXiv Detail & Related papers (2023-08-03T13:08:09Z) - Multi-Task Learning with Prior Information [5.770309971945476]
We propose a multi-task learning framework, where we utilize prior knowledge about the relations between features.
We also impose a penalty on the coefficients changing for each specific feature to ensure related tasks have similar coefficients on common features shared among them.
arXiv Detail & Related papers (2023-01-04T12:48:05Z) - Multi-task Bias-Variance Trade-off Through Functional Constraints [102.64082402388192]
Multi-task learning aims to acquire a set of functions that perform well for diverse tasks.
In this paper we draw intuition from the two extreme learning scenarios -- a single function for all tasks, and a task-specific function that ignores the other tasks.
We introduce a constrained learning formulation that enforces domain specific solutions to a central function.
arXiv Detail & Related papers (2022-10-27T16:06:47Z) - Deep transfer learning for partial differential equations under
conditional shift with DeepONet [0.0]
We propose a novel TL framework for task-specific learning under conditional shift with a deep operator network (DeepONet)
Inspired by the conditional embedding operator theory, we measure the statistical distance between the source domain and the target feature domain.
We show that the proposed TL framework enables fast and efficient multi-task operator learning, despite significant differences between the source and target domains.
arXiv Detail & Related papers (2022-04-20T23:23:38Z) - Multi-task Learning by Leveraging the Semantic Information [14.397128692867799]
We propose to leverage the label information in multi-task learning by exploring the semantic conditional relations among tasks.
Our analysis also leads to a concrete algorithm that jointly matches the semantic distribution and controls label distribution divergence.
arXiv Detail & Related papers (2021-03-03T17:36:35Z) - Incremental Embedding Learning via Zero-Shot Translation [65.94349068508863]
Current state-of-the-art incremental learning methods tackle catastrophic forgetting problem in traditional classification networks.
We propose a novel class-incremental method for embedding network, named as zero-shot translation class-incremental method (ZSTCI)
In addition, ZSTCI can easily be combined with existing regularization-based incremental learning methods to further improve performance of embedding networks.
arXiv Detail & Related papers (2020-12-31T08:21:37Z) - Multi-task Supervised Learning via Cross-learning [102.64082402388192]
We consider a problem known as multi-task learning, consisting of fitting a set of regression functions intended for solving different tasks.
In our novel formulation, we couple the parameters of these functions, so that they learn in their task specific domains while staying close to each other.
This facilitates cross-fertilization in which data collected across different domains help improving the learning performance at each other task.
arXiv Detail & Related papers (2020-10-24T21:35:57Z) - Towards Accurate Knowledge Transfer via Target-awareness Representation
Disentanglement [56.40587594647692]
We propose a novel transfer learning algorithm, introducing the idea of Target-awareness REpresentation Disentanglement (TRED)
TRED disentangles the relevant knowledge with respect to the target task from the original source model and used as a regularizer during fine-tuning the target model.
Experiments on various real world datasets show that our method stably improves the standard fine-tuning by more than 2% in average.
arXiv Detail & Related papers (2020-10-16T17:45:08Z) - Unsupervised Transfer Learning for Spatiotemporal Predictive Networks [90.67309545798224]
We study how to transfer knowledge from a zoo of unsupervisedly learned models towards another network.
Our motivation is that models are expected to understand complex dynamics from different sources.
Our approach yields significant improvements on three benchmarks fortemporal prediction, and benefits the target even from less relevant ones.
arXiv Detail & Related papers (2020-09-24T15:40:55Z) - Multitask learning over graphs: An Approach for Distributed, Streaming
Machine Learning [46.613346075513206]
Multitask learning is an approach to inductive transfer learning.
Recent years have witnessed an increasing ability to collect data in a distributed and streaming manner.
This requires the design of new strategies for learning jointly multiple tasks from streaming data over distributed (or networked) systems.
arXiv Detail & Related papers (2020-01-07T15:32:57Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.