Self-training Avoids Using Spurious Features Under Domain Shift
- URL: http://arxiv.org/abs/2006.10032v3
- Date: Mon, 7 Dec 2020 19:27:22 GMT
- Title: Self-training Avoids Using Spurious Features Under Domain Shift
- Authors: Yining Chen, Colin Wei, Ananya Kumar, Tengyu Ma
- Abstract summary: In unsupervised domain adaptation, conditional entropy minimization and pseudo-labeling work even when the domain shifts are much larger than those analyzed by existing theory.
We identify and analyze one particular setting where the domain shift can be large, but certain spurious features correlate with label in the source domain but are independent label in the target.
- Score: 54.794607791641745
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In unsupervised domain adaptation, existing theory focuses on situations
where the source and target domains are close. In practice, conditional entropy
minimization and pseudo-labeling work even when the domain shifts are much
larger than those analyzed by existing theory. We identify and analyze one
particular setting where the domain shift can be large, but these algorithms
provably work: certain spurious features correlate with the label in the source
domain but are independent of the label in the target. Our analysis considers
linear classification where the spurious features are Gaussian and the
non-spurious features are a mixture of log-concave distributions. For this
setting, we prove that entropy minimization on unlabeled target data will avoid
using the spurious feature if initialized with a decently accurate source
classifier, even though the objective is non-convex and contains multiple bad
local minima using the spurious features. We verify our theory for spurious
domain shift tasks on semi-synthetic Celeb-A and MNIST datasets. Our results
suggest that practitioners collect and self-train on large, diverse datasets to
reduce biases in classifiers even if labeling is impractical.
Related papers
- Domain Adaptation Using Pseudo Labels [16.79672078512152]
In the absence of labeled target data, unsupervised domain adaptation approaches seek to align the marginal distributions of the source and target domains.
We deploy a pretrained network to determine accurate labels for the target domain using a multi-stage pseudo-label refinement procedure.
Our results on multiple datasets demonstrate the effectiveness of our simple procedure in comparison with complex state-of-the-art techniques.
arXiv Detail & Related papers (2024-02-09T22:15:11Z) - Out-Of-Domain Unlabeled Data Improves Generalization [0.7589678255312519]
We propose a novel framework for incorporating unlabeled data into semi-supervised classification problems.
We show that unlabeled samples can be harnessed to narrow the generalization gap.
We validate our claims through experiments conducted on a variety of synthetic and real-world datasets.
arXiv Detail & Related papers (2023-09-29T02:00:03Z) - Adapting to Latent Subgroup Shifts via Concepts and Proxies [82.01141290360562]
We show that the optimal target predictor can be non-parametrically identified with the help of concept and proxy variables available only in the source domain.
For continuous observations, we propose a latent variable model specific to the data generation process at hand.
arXiv Detail & Related papers (2022-12-21T18:30:22Z) - Robust Target Training for Multi-Source Domain Adaptation [110.77704026569499]
We propose a novel Bi-level Optimization based Robust Target Training (BORT$2$) method for MSDA.
Our proposed method achieves the state of the art performance on three MSDA benchmarks, including the large-scale DomainNet dataset.
arXiv Detail & Related papers (2022-10-04T15:20:01Z) - Semantic Concentration for Domain Adaptation [23.706231329913113]
Domain adaptation (DA) paves the way for label annotation and dataset bias issues by the knowledge transfer from a label-rich source domain to a related but unlabeled target domain.
A mainstream of DA methods is to align the feature distributions of the two domains.
We propose Semantic Concentration for Domain Adaptation to encourage the model to concentrate on the most principal features.
arXiv Detail & Related papers (2021-08-12T13:04:36Z) - Instance Level Affinity-Based Transfer for Unsupervised Domain
Adaptation [74.71931918541748]
We propose an instance affinity based criterion for source to target transfer during adaptation, called ILA-DA.
We first propose a reliable and efficient method to extract similar and dissimilar samples across source and target, and utilize a multi-sample contrastive loss to drive the domain alignment process.
We verify the effectiveness of ILA-DA by observing consistent improvements in accuracy over popular domain adaptation approaches on a variety of benchmark datasets.
arXiv Detail & Related papers (2021-04-03T01:33:14Z) - A Free Lunch for Unsupervised Domain Adaptive Object Detection without
Source Data [69.091485888121]
Unsupervised domain adaptation assumes that source and target domain data are freely available and usually trained together to reduce the domain gap.
We propose a source data-free domain adaptive object detection (SFOD) framework via modeling it into a problem of learning with noisy labels.
arXiv Detail & Related papers (2020-12-10T01:42:35Z) - Learning Target Domain Specific Classifier for Partial Domain Adaptation [85.71584004185031]
Unsupervised domain adaptation (UDA) aims at reducing the distribution discrepancy when transferring knowledge from a labeled source domain to an unlabeled target domain.
This paper focuses on a more realistic UDA scenario, where the target label space is subsumed to the source label space.
arXiv Detail & Related papers (2020-08-25T02:28:24Z) - MiniMax Entropy Network: Learning Category-Invariant Features for Domain Adaptation [29.43532067090422]
We propose an easy-to-implement method dubbed MiniMax Entropy Networks (MMEN) based on adversarial learning.
Unlike most existing approaches which employ a generator to deal with domain difference, MMEN focuses on learning the categorical information from unlabeled target samples.
arXiv Detail & Related papers (2019-04-21T13:39:29Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.