Algorithms and Theory for Supervised Gradual Domain Adaptation
- URL: http://arxiv.org/abs/2204.11644v1
- Date: Mon, 25 Apr 2022 13:26:11 GMT
- Title: Algorithms and Theory for Supervised Gradual Domain Adaptation
- Authors: Jing Dong, Shiji Zhou, Baoxiang Wang, Han Zhao
- Abstract summary: We study the problem of supervised gradual domain adaptation, where labeled data from shifting distributions are available to the learner along the trajectory.
Under this setting, we provide the first generalization upper bound on the learning error under mild assumptions.
Our results are algorithm agnostic for a range of loss functions, and only depend linearly on the averaged learning error across the trajectory.
- Score: 19.42476993856205
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The phenomenon of data distribution evolving over time has been observed in a
range of applications, calling the needs of adaptive learning algorithms. We
thus study the problem of supervised gradual domain adaptation, where labeled
data from shifting distributions are available to the learner along the
trajectory, and we aim to learn a classifier on a target data distribution of
interest. Under this setting, we provide the first generalization upper bound
on the learning error under mild assumptions. Our results are algorithm
agnostic, general for a range of loss functions, and only depend linearly on
the averaged learning error across the trajectory. This shows significant
improvement compared to the previous upper bound for unsupervised gradual
domain adaptation, where the learning error on the target domain depends
exponentially on the initial error on the source domain. Compared with the
offline setting of learning from multiple domains, our results also suggest the
potential benefits of the temporal structure among different domains in
adapting to the target one. Empirically, our theoretical results imply that
learning proper representations across the domains will effectively mitigate
the learning errors. Motivated by these theoretical insights, we propose a
min-max learning objective to learn the representation and classifier
simultaneously. Experimental results on both semi-synthetic and large-scale
real datasets corroborate our findings and demonstrate the effectiveness of our
objectives.
Related papers
- Non-stationary Domain Generalization: Theory and Algorithm [11.781050299571692]
In this paper, we study domain generalization in non-stationary environment.
We first examine the impact of environmental non-stationarity on model performance.
Then, we propose a novel algorithm based on adaptive invariant representation learning.
arXiv Detail & Related papers (2024-05-10T21:32:43Z) - Domain Adaptation Principal Component Analysis: base linear method for
learning with out-of-distribution data [55.41644538483948]
Domain adaptation is a popular paradigm in modern machine learning.
We present a method called Domain Adaptation Principal Component Analysis (DAPCA)
DAPCA finds a linear reduced data representation useful for solving the domain adaptation task.
arXiv Detail & Related papers (2022-08-28T21:10:56Z) - Self-balanced Learning For Domain Generalization [64.99791119112503]
Domain generalization aims to learn a prediction model on multi-domain source data such that the model can generalize to a target domain with unknown statistics.
Most existing approaches have been developed under the assumption that the source data is well-balanced in terms of both domain and class.
We propose a self-balanced domain generalization framework that adaptively learns the weights of losses to alleviate the bias caused by different distributions of the multi-domain source data.
arXiv Detail & Related papers (2021-08-31T03:17:54Z) - f-Domain-Adversarial Learning: Theory and Algorithms [82.97698406515667]
Unsupervised domain adaptation is used in many machine learning applications where, during training, a model has access to unlabeled data in the target domain.
We derive a novel generalization bound for domain adaptation that exploits a new measure of discrepancy between distributions based on a variational characterization of f-divergences.
arXiv Detail & Related papers (2021-06-21T18:21:09Z) - Learning a Domain-Agnostic Visual Representation for Autonomous Driving
via Contrastive Loss [25.798361683744684]
Domain-Agnostic Contrastive Learning (DACL) is a two-stage unsupervised domain adaptation framework with cyclic adversarial training and contrastive loss.
Our proposed approach achieves better performance in the monocular depth estimation task compared to previous state-of-the-art methods.
arXiv Detail & Related papers (2021-03-10T07:06:03Z) - Adaptive Risk Minimization: Learning to Adapt to Domain Shift [109.87561509436016]
A fundamental assumption of most machine learning algorithms is that the training and test data are drawn from the same underlying distribution.
In this work, we consider the problem setting of domain generalization, where the training data are structured into domains and there may be multiple test time shifts.
We introduce the framework of adaptive risk minimization (ARM), in which models are directly optimized for effective adaptation to shift by learning to adapt on the training domains.
arXiv Detail & Related papers (2020-07-06T17:59:30Z) - Progressive Graph Learning for Open-Set Domain Adaptation [48.758366879597965]
Domain shift is a fundamental problem in visual recognition which typically arises when the source and target data follow different distributions.
In this paper, we tackle a more realistic problem of open-set domain shift where the target data contains additional classes that are not present in the source data.
We introduce an end-to-end Progressive Graph Learning framework where a graph neural network with episodic training is integrated to suppress underlying conditional shift.
arXiv Detail & Related papers (2020-06-22T09:10:34Z) - A survey on domain adaptation theory: learning bounds and theoretical
guarantees [17.71634393160982]
The main objective of this survey is to provide an overview of the state-of-the-art theoretical results in a specific, and arguably the most popular, sub-field of transfer learning.
In this sub-field, the data distribution is assumed to change across the training and the test data, while the learning task remains the same.
We provide a first up-to-date description of existing results related to domain adaptation problem.
arXiv Detail & Related papers (2020-04-24T16:11:03Z) - Understanding Self-Training for Gradual Domain Adaptation [107.37869221297687]
We consider gradual domain adaptation, where the goal is to adapt an initial classifier trained on a source domain given only unlabeled data that shifts gradually in distribution towards a target domain.
We prove the first non-vacuous upper bound on the error of self-training with gradual shifts, under settings where directly adapting to the target domain can result in unbounded error.
The theoretical analysis leads to algorithmic insights, highlighting that regularization and label sharpening are essential even when we have infinite data, and suggesting that self-training works particularly well for shifts with small Wasserstein-infinity distance.
arXiv Detail & Related papers (2020-02-26T08:59:40Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.