ZIN: When and How to Learn Invariance by Environment Inference?

• URL: http://arxiv.org/abs/2203.05818v1
• Date: Fri, 11 Mar 2022 10:00:33 GMT
• Title: ZIN: When and How to Learn Invariance by Environment Inference?
• Authors: Yong Lin, Shengyu Zhu, Peng Cui
• Abstract summary: Invariant learning methods have proposed to learn robust and invariant models based on environment partition. We show that learning invariant features under this circumstance is fundamentally impossible without further inductive biases or additional information. We propose a framework to jointly learn environment partition and invariant representation, assisted by additional auxiliary information.
• Score: 24.191152823045385
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: It is commonplace to encounter heterogeneous data, of which some aspects of the data distribution may vary but the underlying causal mechanisms remain constant. When data are divided into distinct environments according to the heterogeneity, recent invariant learning methods have proposed to learn robust and invariant models based on this environment partition. It is hence tempting to utilize the inherent heterogeneity even when environment partition is not provided. Unfortunately, in this work, we show that learning invariant features under this circumstance is fundamentally impossible without further inductive biases or additional information. Then, we propose a framework to jointly learn environment partition and invariant representation, assisted by additional auxiliary information. We derive sufficient and necessary conditions for our framework to provably identify invariant features under a fairly general setting. Experimental results on both synthetic and real world datasets validate our analysis and demonstrate an improved performance of the proposed framework over existing methods. Finally, our results also raise the need of making the role of inductive biases more explicit in future works, when considering learning invariant models without environment partition.

Related papers

• Mining Invariance from Nonlinear Multi-Environment Data: Binary Classification [2.0528878959274883]
  This paper focuses on binary classification to shed light on general nonlinear data generation mechanisms. We identify a unique form of invariance that exists solely in a binary setting that allows us to train models invariant over environments. We propose a prediction method and conduct experiments using real and synthetic datasets.
  arXiv  Detail & Related papers  (2024-04-23T17:26:59Z)
• Flow Factorized Representation Learning [109.51947536586677]
  We introduce a generative model which specifies a distinct set of latent probability paths that define different input transformations. We show that our model achieves higher likelihoods on standard representation learning benchmarks while simultaneously being closer to approximately equivariant models.
  arXiv  Detail & Related papers  (2023-09-22T20:15:37Z)
• Conformal Inference for Invariant Risk Minimization [12.049545417799125]
  The application of machine learning models can be significantly impeded by the occurrence of distributional shifts. One way to tackle this problem is to use invariant learning, such as invariant risk minimization (IRM), to acquire an invariant representation. This paper develops methods for obtaining distribution-free prediction regions to describe uncertainty estimates for invariant representations.
  arXiv  Detail & Related papers  (2023-05-22T03:48:38Z)
• Decorr: Environment Partitioning for Invariant Learning and OOD Generalization [10.799855921851332]
  Invariant learning methods are aimed at identifying a consistent predictor across multiple environments. When environments aren't inherent in the data, practitioners must define them manually. This environment partitioning affects invariant learning's efficacy but remains underdiscussed. In this paper, we suggest partitioning the dataset into several environments by isolating low-correlation data subsets.
  arXiv  Detail & Related papers  (2022-11-18T06:49:35Z)
• Equivariant Disentangled Transformation for Domain Generalization under Combination Shift [91.38796390449504]
  Combinations of domains and labels are not observed during training but appear in the test environment. We provide a unique formulation of the combination shift problem based on the concepts of homomorphism, equivariance, and a refined definition of disentanglement.
  arXiv  Detail & Related papers  (2022-08-03T12:31:31Z)
• Predicting Out-of-Domain Generalization with Neighborhood Invariance [59.05399533508682]
  We propose a measure of a classifier's output invariance in a local transformation neighborhood. Our measure is simple to calculate, does not depend on the test point's true label, and can be applied even in out-of-domain (OOD) settings. In experiments on benchmarks in image classification, sentiment analysis, and natural language inference, we demonstrate a strong and robust correlation between our measure and actual OOD generalization.
  arXiv  Detail & Related papers  (2022-07-05T14:55:16Z)
• Differentiable Invariant Causal Discovery [106.87950048845308]
  Learning causal structure from observational data is a fundamental challenge in machine learning. This paper proposes Differentiable Invariant Causal Discovery (DICD) to avoid learning spurious edges and wrong causal directions. Extensive experiments on synthetic and real-world datasets verify that DICD outperforms state-of-the-art causal discovery methods up to 36% in SHD.
  arXiv  Detail & Related papers  (2022-05-31T09:29:07Z)
• Equivariance Allows Handling Multiple Nuisance Variables When Analyzing Pooled Neuroimaging Datasets [53.34152466646884]
  In this paper, we show how bringing recent results on equivariant representation learning instantiated on structured spaces together with simple use of classical results on causal inference provides an effective practical solution. We demonstrate how our model allows dealing with more than one nuisance variable under some assumptions and can enable analysis of pooled scientific datasets in scenarios that would otherwise entail removing a large portion of the samples.
  arXiv  Detail & Related papers  (2022-03-29T04:54:06Z)
• Balancing Fairness and Robustness via Partial Invariance [17.291131923335918]
  Invariant Risk Minimization (IRM) aims to learn invariant features from a set of environments for solving the out-of-distribution (OOD) problem. We argue for a partial invariance framework to mitigate the failure case. Our results show the capability of the partial invariant risk minimization to alleviate the trade-off between fairness and risk in certain settings.
  arXiv  Detail & Related papers  (2021-12-17T06:41:47Z)
• Learning Conditional Invariance through Cycle Consistency [60.85059977904014]
  We propose a novel approach to identify meaningful and independent factors of variation in a dataset. Our method involves two separate latent subspaces for the target property and the remaining input information. We demonstrate on synthetic and molecular data that our approach identifies more meaningful factors which lead to sparser and more interpretable models.
  arXiv  Detail & Related papers  (2021-11-25T17:33:12Z)
• Environment Inference for Invariant Learning [9.63004099102596]
  We propose EIIL, a framework for domain-invariant learning that incorporates Environment Inference. We show that EIIL outperforms invariant learning methods on the CMNIST benchmark without using environment labels. We also establish connections between EIIL and algorithmic fairness, which enables EIIL to improve accuracy and calibration in a fair prediction problem.
  arXiv  Detail & Related papers  (2020-10-14T17:11:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.