Invariant Correlation of Representation with Label

• URL: http://arxiv.org/abs/2407.01749v1
• Date: Mon, 1 Jul 2024 19:27:28 GMT
• Title: Invariant Correlation of Representation with Label
• Authors: Gaojie Jin, Ronghui Mu, Xinping Yi, Xiaowei Huang, Lijun Zhang,
• Abstract summary: We introduce ICorr (an abbreviation for textbfInvariant textbfCorrelation), a novel approach designed to surmount the challenge of domain generalization in noisy settings. We empirically demonstrate the effectiveness of ICorr by comparing it with other domain generalization methods on various noisy datasets.
• Score: 30.82552387894663
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Invariant Risk Minimization (IRM) approach aims to address the challenge of domain generalization by training a feature representation that remains invariant across multiple environments. However, in noisy environments, IRM-related techniques such as IRMv1 and VREx may be unable to achieve the optimal IRM solution, primarily due to erroneous optimization directions. To address this issue, we introduce ICorr (an abbreviation for \textbf{I}nvariant \textbf{Corr}elation), a novel approach designed to surmount the above challenge in noisy settings. Additionally, we dig into a case study to analyze why previous methods may lose ground while ICorr can succeed. Through a theoretical lens, particularly from a causality perspective, we illustrate that the invariant correlation of representation with label is a necessary condition for the optimal invariant predictor in noisy environments, whereas the optimization motivations for other methods may not be. Furthermore, we empirically demonstrate the effectiveness of ICorr by comparing it with other domain generalization methods on various noisy datasets.

Related papers

• From Inverse Optimization to Feasibility to ERM [11.731853838892487]
  We study the contextual inverse setting that utilizes additional contextual information to better predict parameters. We experimentally validate our approach on synthetic and real-world problems and demonstrate improved performance compared to existing methods.
  arXiv  Detail & Related papers  (2024-02-27T21:06:42Z)
• Continual Invariant Risk Minimization [46.051656238770086]
  Empirical risk minimization can lead to poor generalization behavior on unseen environments if the learned model does not capture invariant feature representations. Invariant risk minimization (IRM) is a recent proposal for discovering environment-invariant representations.
  arXiv  Detail & Related papers  (2023-10-21T11:44:47Z)
• Exploiting Temporal Structures of Cyclostationary Signals for Data-Driven Single-Channel Source Separation [98.95383921866096]
  We study the problem of single-channel source separation (SCSS) We focus on cyclostationary signals, which are particularly suitable in a variety of application domains. We propose a deep learning approach using a U-Net architecture, which is competitive with the minimum MSE estimator.
  arXiv  Detail & Related papers  (2022-08-22T14:04:56Z)
• Repeated Environment Inference for Invariant Learning [8.372465442144046]
  We focus on the invariant representation notion when the Bayes optimal conditional label distribution is the same across different environments. Previous work conducts Environment Inference (EI) by maximizing the penalty term from Invariant Risk Minimization (IRM) framework. We show that this method outperforms baselines on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2022-07-26T13:07:22Z)
• Optimizing Information-theoretical Generalization Bounds via Anisotropic Noise in SGLD [73.55632827932101]
  We optimize the information-theoretical generalization bound by manipulating the noise structure in SGLD. We prove that with constraint to guarantee low empirical risk, the optimal noise covariance is the square root of the expected gradient covariance.
  arXiv  Detail & Related papers  (2021-10-26T15:02:27Z)
• Environment Inference for Invariant Learning [9.63004099102596]
  We propose EIIL, a framework for domain-invariant learning that incorporates Environment Inference. We show that EIIL outperforms invariant learning methods on the CMNIST benchmark without using environment labels. We also establish connections between EIIL and algorithmic fairness, which enables EIIL to improve accuracy and calibration in a fair prediction problem.
  arXiv  Detail & Related papers  (2020-10-14T17:11:46Z)
• The Risks of Invariant Risk Minimization [52.7137956951533]
  Invariant Risk Minimization is an objective based on the idea for learning deep, invariant features of data. We present the first analysis of classification under the IRM objective--as well as these recently proposed alternatives--under a fairly natural and general model. We show that IRM can fail catastrophically unless the test data are sufficiently similar to the training distribution--this is precisely the issue that it was intended to solve.
  arXiv  Detail & Related papers  (2020-10-12T14:54:32Z)
• Learning Invariant Representations and Risks for Semi-supervised Domain Adaptation [109.73983088432364]
  We propose the first method that aims to simultaneously learn invariant representations and risks under the setting of semi-supervised domain adaptation (Semi-DA) We introduce the LIRR algorithm for jointly textbfLearning textbfInvariant textbfRepresentations and textbfRisks.
  arXiv  Detail & Related papers  (2020-10-09T15:42:35Z)
• An Empirical Study of Invariant Risk Minimization [5.412466703928342]
  Invariant risk minimization is a proposed framework for learning predictors that are invariant to spurious correlations. Despite its theoretical justifications, IRM has not been extensively tested across various settings. We empirically investigate several research questions using IRMv1, which is the first practical algorithm proposed to approximately solve IRM.
  arXiv  Detail & Related papers  (2020-04-10T12:23:29Z)
• Invariant Causal Prediction for Block MDPs [106.63346115341862]
  Generalization across environments is critical to the successful application of reinforcement learning algorithms to real-world challenges. We propose a method of invariant prediction to learn model-irrelevance state abstractions (MISA) that generalize to novel observations in the multi-environment setting.
  arXiv  Detail & Related papers  (2020-03-12T21:03:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.