Provable Domain Generalization via Invariant-Feature Subspace Recovery
- URL: http://arxiv.org/abs/2201.12919v1
- Date: Sun, 30 Jan 2022 21:22:47 GMT
- Title: Provable Domain Generalization via Invariant-Feature Subspace Recovery
- Authors: Haoxiang Wang, Haozhe Si, Bo Li, Han Zhao
- Abstract summary: In this paper, we propose to achieve domain generalization with Invariant- Subspace Recovery (ISR)
Unlike training IRM, our algorithms bypass non-variantity issues and enjoy global convergence.
In addition, on three real-world image datasets, we show that ISR- can be used as a simple yet effective post-processing method.
- Score: 18.25619572103648
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Domain generalization asks for models trained on a set of training
environments to perform well on unseen test environments. Recently, a series of
algorithms such as Invariant Risk Minimization (IRM) has been proposed for
domain generalization. However, Rosenfeld et al. (2021) shows that in a simple
linear data model, even if non-convexity issues are ignored, IRM and its
extensions cannot generalize to unseen environments with less than $d_s+1$
training environments, where $d_s$ is the dimension of the spurious-feature
subspace. In this paper, we propose to achieve domain generalization with
Invariant-feature Subspace Recovery (ISR). Our first algorithm, ISR-Mean, can
identify the subspace spanned by invariant features from the first-order
moments of the class-conditional distributions, and achieve provable domain
generalization with $d_s+1$ training environments under the data model of
Rosenfeld et al. (2021). Our second algorithm, ISR-Cov, further reduces the
required number of training environments to $O(1)$ using the information of
second-order moments. Notably, unlike IRM, our algorithms bypass non-convexity
issues and enjoy global convergence guarantees. Empirically, our ISRs can
obtain superior performance compared with IRM on synthetic benchmarks. In
addition, on three real-world image and text datasets, we show that ISR-Mean
can be used as a simple yet effective post-processing method to increase the
worst-case accuracy of trained models against spurious correlations and group
shifts.
Related papers
- Invariant-Feature Subspace Recovery: A New Class of Provable Domain
Generalization Algorithms [14.248005245508432]
Domain generalization asks for trained models over a set of training environments to generalize well in unseen test environments.
We propose Subspace Recovery (ISR): a new class of algorithms to achieve provable regression problems.
ISR can be used as post-processing methods for neural nets such as neural nets Empirically, we demonstrate the superior performance of our ISRs on synthetic benchmarks.
arXiv Detail & Related papers (2023-11-02T03:24:55Z) - Out-Of-Domain Unlabeled Data Improves Generalization [0.7589678255312519]
We propose a novel framework for incorporating unlabeled data into semi-supervised classification problems.
We show that unlabeled samples can be harnessed to narrow the generalization gap.
We validate our claims through experiments conducted on a variety of synthetic and real-world datasets.
arXiv Detail & Related papers (2023-09-29T02:00:03Z) - One-Shot Domain Adaptive and Generalizable Semantic Segmentation with
Class-Aware Cross-Domain Transformers [96.51828911883456]
Unsupervised sim-to-real domain adaptation (UDA) for semantic segmentation aims to improve the real-world test performance of a model trained on simulated data.
Traditional UDA often assumes that there are abundant unlabeled real-world data samples available during training for the adaptation.
We explore the one-shot unsupervised sim-to-real domain adaptation (OSUDA) and generalization problem, where only one real-world data sample is available.
arXiv Detail & Related papers (2022-12-14T15:54:15Z) - A Relational Intervention Approach for Unsupervised Dynamics
Generalization in Model-Based Reinforcement Learning [113.75991721607174]
We introduce an interventional prediction module to estimate the probability of two estimated $hatz_i, hatz_j$ belonging to the same environment.
We empirically show that $hatZ$ estimated by our method enjoy less redundant information than previous methods.
arXiv Detail & Related papers (2022-06-09T15:01:36Z) - Iterative Feature Matching: Toward Provable Domain Generalization with
Logarithmic Environments [55.24895403089543]
Domain generalization aims at performing well on unseen test environments with data from a limited number of training environments.
We present a new algorithm based on performing iterative feature matching that is guaranteed with high probability to yield a predictor that generalizes after seeing only $O(logd_s)$ environments.
arXiv Detail & Related papers (2021-06-18T04:39:19Z) - Adversarially Adaptive Normalization for Single Domain Generalization [71.80587939738672]
We propose a generic normalization approach, adaptive standardization and rescaling normalization (ASR-Norm)
ASR-Norm learns both the standardization and rescaling statistics via neural networks.
We show that ASR-Norm can bring consistent improvement to the state-of-the-art ADA approaches.
arXiv Detail & Related papers (2021-06-01T23:58:23Z) - The Risks of Invariant Risk Minimization [52.7137956951533]
Invariant Risk Minimization is an objective based on the idea for learning deep, invariant features of data.
We present the first analysis of classification under the IRM objective--as well as these recently proposed alternatives--under a fairly natural and general model.
We show that IRM can fail catastrophically unless the test data are sufficiently similar to the training distribution--this is precisely the issue that it was intended to solve.
arXiv Detail & Related papers (2020-10-12T14:54:32Z) - Learning Invariant Representations and Risks for Semi-supervised Domain
Adaptation [109.73983088432364]
We propose the first method that aims to simultaneously learn invariant representations and risks under the setting of semi-supervised domain adaptation (Semi-DA)
We introduce the LIRR algorithm for jointly textbfLearning textbfInvariant textbfRepresentations and textbfRisks.
arXiv Detail & Related papers (2020-10-09T15:42:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.