Decorr: Environment Partitioning for Invariant Learning and OOD Generalization
- URL: http://arxiv.org/abs/2211.10054v2
- Date: Wed, 22 May 2024 08:34:24 GMT
- Title: Decorr: Environment Partitioning for Invariant Learning and OOD Generalization
- Authors: Yufan Liao, Qi Wu, Zhaodi Wu, Xing Yan,
- Abstract summary: Invariant learning methods are aimed at identifying a consistent predictor across multiple environments.
When environments aren't inherent in the data, practitioners must define them manually.
This environment partitioning affects invariant learning's efficacy but remains underdiscussed.
In this paper, we suggest partitioning the dataset into several environments by isolating low-correlation data subsets.
- Score: 10.799855921851332
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: Invariant learning methods, aimed at identifying a consistent predictor across multiple environments, are gaining prominence in out-of-distribution (OOD) generalization. Yet, when environments aren't inherent in the data, practitioners must define them manually. This environment partitioning--algorithmically segmenting the training dataset into environments--crucially affects invariant learning's efficacy but remains underdiscussed. Proper environment partitioning could broaden the applicability of invariant learning and enhance its performance. In this paper, we suggest partitioning the dataset into several environments by isolating low-correlation data subsets. Through experiments with synthetic and real data, our Decorr method demonstrates superior performance in combination with invariant learning. Decorr mitigates the issue of spurious correlations, aids in identifying stable predictors, and broadens the applicability of invariant learning methods.
Related papers
- CGLearn: Consistent Gradient-Based Learning for Out-of-Distribution Generalization [0.7366405857677226]
In this work, we introduce a simple yet powerful approach, CGLearn, which relies on the agreement of gradients across various environments.
Our proposed method demonstrates superior performance compared to state-of-the-art methods in both linear and nonlinear settings.
Comprehensive experiments on both synthetic and real-world datasets highlight its effectiveness in diverse scenarios.
arXiv Detail & Related papers (2024-11-09T02:36:39Z) - The Implicit Bias of Heterogeneity towards Invariance: A Study of Multi-Environment Matrix Sensing [9.551225697705199]
This paper studies the implicit bias of Gradient Descent (SGD) over heterogeneous data and shows that the implicit bias drives the model learning towards an invariant solution.
Specifically, we theoretically investigate the multi-environment low-rank matrix sensing problem where in each environment, the signal comprises (i) a lower-rank invariant part shared across all environments; and (ii) a significantly varying environment-dependent spurious component.
The key insight is, through simply employing the large step size large-batch SGD sequentially in each environment without any explicit regularization, the oscillation caused by heterogeneity can provably prevent model learning spurious signals.
arXiv Detail & Related papers (2024-03-03T07:38:24Z) - Graph Invariant Learning with Subgraph Co-mixup for Out-Of-Distribution
Generalization [51.913685334368104]
We propose a novel graph invariant learning method based on invariant and variant patterns co-mixup strategy.
Our method significantly outperforms state-of-the-art under various distribution shifts.
arXiv Detail & Related papers (2023-12-18T07:26:56Z) - Conformal Inference for Invariant Risk Minimization [12.049545417799125]
The application of machine learning models can be significantly impeded by the occurrence of distributional shifts.
One way to tackle this problem is to use invariant learning, such as invariant risk minimization (IRM), to acquire an invariant representation.
This paper develops methods for obtaining distribution-free prediction regions to describe uncertainty estimates for invariant representations.
arXiv Detail & Related papers (2023-05-22T03:48:38Z) - Boosting Differentiable Causal Discovery via Adaptive Sample Reweighting [62.23057729112182]
Differentiable score-based causal discovery methods learn a directed acyclic graph from observational data.
We propose a model-agnostic framework to boost causal discovery performance by dynamically learning the adaptive weights for the Reweighted Score function, ReScore.
arXiv Detail & Related papers (2023-03-06T14:49:59Z) - Unleashing the Power of Graph Data Augmentation on Covariate
Distribution Shift [50.98086766507025]
We propose a simple-yet-effective data augmentation strategy, Adversarial Invariant Augmentation (AIA)
AIA aims to extrapolate and generate new environments, while concurrently preserving the original stable features during the augmentation process.
arXiv Detail & Related papers (2022-11-05T07:55:55Z) - Differentiable Invariant Causal Discovery [106.87950048845308]
Learning causal structure from observational data is a fundamental challenge in machine learning.
This paper proposes Differentiable Invariant Causal Discovery (DICD) to avoid learning spurious edges and wrong causal directions.
Extensive experiments on synthetic and real-world datasets verify that DICD outperforms state-of-the-art causal discovery methods up to 36% in SHD.
arXiv Detail & Related papers (2022-05-31T09:29:07Z) - ZIN: When and How to Learn Invariance by Environment Inference? [24.191152823045385]
Invariant learning methods have proposed to learn robust and invariant models based on environment partition.
We show that learning invariant features under this circumstance is fundamentally impossible without further inductive biases or additional information.
We propose a framework to jointly learn environment partition and invariant representation, assisted by additional auxiliary information.
arXiv Detail & Related papers (2022-03-11T10:00:33Z) - Learning Conditional Invariance through Cycle Consistency [60.85059977904014]
We propose a novel approach to identify meaningful and independent factors of variation in a dataset.
Our method involves two separate latent subspaces for the target property and the remaining input information.
We demonstrate on synthetic and molecular data that our approach identifies more meaningful factors which lead to sparser and more interpretable models.
arXiv Detail & Related papers (2021-11-25T17:33:12Z) - Predict then Interpolate: A Simple Algorithm to Learn Stable Classifiers [59.06169363181417]
Predict then Interpolate (PI) is an algorithm for learning correlations that are stable across environments.
We prove that by interpolating the distributions of the correct predictions and the wrong predictions, we can uncover an oracle distribution where the unstable correlation vanishes.
arXiv Detail & Related papers (2021-05-26T15:37:48Z) - Environment Inference for Invariant Learning [9.63004099102596]
We propose EIIL, a framework for domain-invariant learning that incorporates Environment Inference.
We show that EIIL outperforms invariant learning methods on the CMNIST benchmark without using environment labels.
We also establish connections between EIIL and algorithmic fairness, which enables EIIL to improve accuracy and calibration in a fair prediction problem.
arXiv Detail & Related papers (2020-10-14T17:11:46Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.