Provable Domain Generalization via Invariant-Feature Subspace Recovery
- URL: http://arxiv.org/abs/2201.12919v1
- Date: Sun, 30 Jan 2022 21:22:47 GMT
- Title: Provable Domain Generalization via Invariant-Feature Subspace Recovery
- Authors: Haoxiang Wang, Haozhe Si, Bo Li, Han Zhao
- Abstract summary: In this paper, we propose to achieve domain generalization with Invariant- Subspace Recovery (ISR)
Unlike training IRM, our algorithms bypass non-variantity issues and enjoy global convergence.
In addition, on three real-world image datasets, we show that ISR- can be used as a simple yet effective post-processing method.
- Score: 18.25619572103648
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Domain generalization asks for models trained on a set of training
environments to perform well on unseen test environments. Recently, a series of
algorithms such as Invariant Risk Minimization (IRM) has been proposed for
domain generalization. However, Rosenfeld et al. (2021) shows that in a simple
linear data model, even if non-convexity issues are ignored, IRM and its
extensions cannot generalize to unseen environments with less than $d_s+1$
training environments, where $d_s$ is the dimension of the spurious-feature
subspace. In this paper, we propose to achieve domain generalization with
Invariant-feature Subspace Recovery (ISR). Our first algorithm, ISR-Mean, can
identify the subspace spanned by invariant features from the first-order
moments of the class-conditional distributions, and achieve provable domain
generalization with $d_s+1$ training environments under the data model of
Rosenfeld et al. (2021). Our second algorithm, ISR-Cov, further reduces the
required number of training environments to $O(1)$ using the information of
second-order moments. Notably, unlike IRM, our algorithms bypass non-convexity
issues and enjoy global convergence guarantees. Empirically, our ISRs can
obtain superior performance compared with IRM on synthetic benchmarks. In
addition, on three real-world image and text datasets, we show that ISR-Mean
can be used as a simple yet effective post-processing method to increase the
worst-case accuracy of trained models against spurious correlations and group
shifts.
Related papers
- Towards Generalizable Trajectory Prediction Using Dual-Level Representation Learning And Adaptive Prompting [107.4034346788744]
Existing vehicle trajectory prediction models struggle with generalizability, prediction uncertainties, and handling complex interactions.
We propose Perceiver with Register queries (PerReg+), a novel trajectory prediction framework that introduces: (1) Dual-Level Representation Learning via Self-Distillation (SD) and Masked Reconstruction (MR), capturing global context and fine-grained details; (2) Enhanced Multimodality using register-based queries and pretraining, eliminating the need for clustering and suppression; and (3) Adaptive Prompt Tuning during fine-tuning, freezing the main architecture and optimizing a small number of prompts for efficient adaptation.
arXiv Detail & Related papers (2025-01-08T20:11:09Z) - Invariant-Feature Subspace Recovery: A New Class of Provable Domain
Generalization Algorithms [14.248005245508432]
Domain generalization asks for trained models over a set of training environments to generalize well in unseen test environments.
We propose Subspace Recovery (ISR): a new class of algorithms to achieve provable regression problems.
ISR can be used as post-processing methods for neural nets such as neural nets Empirically, we demonstrate the superior performance of our ISRs on synthetic benchmarks.
arXiv Detail & Related papers (2023-11-02T03:24:55Z) - Out-Of-Domain Unlabeled Data Improves Generalization [0.7589678255312519]
We propose a novel framework for incorporating unlabeled data into semi-supervised classification problems.
We show that unlabeled samples can be harnessed to narrow the generalization gap.
We validate our claims through experiments conducted on a variety of synthetic and real-world datasets.
arXiv Detail & Related papers (2023-09-29T02:00:03Z) - ERM++: An Improved Baseline for Domain Generalization [69.80606575323691]
Empirical Risk Minimization (ERM) can outperform most more complex Domain Generalization (DG) methods when properly tuned.
ERM++ improves DG performance by over 5% compared to prior ERM baselines.
arXiv Detail & Related papers (2023-04-04T17:31:15Z) - A Relational Intervention Approach for Unsupervised Dynamics
Generalization in Model-Based Reinforcement Learning [113.75991721607174]
We introduce an interventional prediction module to estimate the probability of two estimated $hatz_i, hatz_j$ belonging to the same environment.
We empirically show that $hatZ$ estimated by our method enjoy less redundant information than previous methods.
arXiv Detail & Related papers (2022-06-09T15:01:36Z) - Iterative Feature Matching: Toward Provable Domain Generalization with
Logarithmic Environments [55.24895403089543]
Domain generalization aims at performing well on unseen test environments with data from a limited number of training environments.
We present a new algorithm based on performing iterative feature matching that is guaranteed with high probability to yield a predictor that generalizes after seeing only $O(logd_s)$ environments.
arXiv Detail & Related papers (2021-06-18T04:39:19Z) - Adversarially Adaptive Normalization for Single Domain Generalization [71.80587939738672]
We propose a generic normalization approach, adaptive standardization and rescaling normalization (ASR-Norm)
ASR-Norm learns both the standardization and rescaling statistics via neural networks.
We show that ASR-Norm can bring consistent improvement to the state-of-the-art ADA approaches.
arXiv Detail & Related papers (2021-06-01T23:58:23Z) - Learning Invariant Representations and Risks for Semi-supervised Domain
Adaptation [109.73983088432364]
We propose the first method that aims to simultaneously learn invariant representations and risks under the setting of semi-supervised domain adaptation (Semi-DA)
We introduce the LIRR algorithm for jointly textbfLearning textbfInvariant textbfRepresentations and textbfRisks.
arXiv Detail & Related papers (2020-10-09T15:42:35Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.