Pareto Invariant Risk Minimization

• URL: http://arxiv.org/abs/2206.07766v1
• Date: Wed, 15 Jun 2022 19:04:02 GMT
• Title: Pareto Invariant Risk Minimization
• Authors: Yongqiang Chen, Kaiwen Zhou, Yatao Bian, Binghui Xie, Kaili Ma, Yonggang Zhang, Han Yang, Bo Han, James Cheng
• Abstract summary: We propose a new optimization scheme for invariant risk minimization (IRM) called PAreto Invariant Risk Minimization (PAIR) We show PAIR can empower the practical IRM variants to overcome the barriers with the original IRM when provided with proper guidance.
• Score: 32.01775861630696
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Despite the success of invariant risk minimization (IRM) in tackling the Out-of-Distribution generalization problem, IRM can compromise the optimality when applied in practice. The practical variants of IRM, e.g., IRMv1, have been shown to have significant gaps with IRM and thus could fail to capture the invariance even in simple problems. Moreover, the optimization procedure in IRMv1 involves two intrinsically conflicting objectives, and often requires careful tuning for the objective weights. To remedy the above issues, we reformulate IRM as a multi-objective optimization problem, and propose a new optimization scheme for IRM, called PAreto Invariant Risk Minimization (PAIR). PAIR can adaptively adjust the optimization direction under the objective conflicts. Furthermore, we show PAIR can empower the practical IRM variants to overcome the barriers with the original IRM when provided with proper guidance. We conduct experiments with ColoredMNIST to confirm our theory and the effectiveness of PAIR.

Related papers

• Continual Invariant Risk Minimization [46.051656238770086]
  Empirical risk minimization can lead to poor generalization behavior on unseen environments if the learned model does not capture invariant feature representations. Invariant risk minimization (IRM) is a recent proposal for discovering environment-invariant representations.
  arXiv  Detail & Related papers  (2023-10-21T11:44:47Z)
• Out-of-Distribution Optimality of Invariant Risk Minimization [20.389816785165333]
  Invariant Risk Minimization (IRM) is considered to be a promising approach to minimize the o.o.d. risk. This paper rigorously proves that a solution to the bi-level optimization problem minimizes the o.o.d. risk under certain conditions.
  arXiv  Detail & Related papers  (2023-07-22T03:31:15Z)
• On the Variance, Admissibility, and Stability of Empirical Risk Minimization [80.26309576810844]
  Empirical Risk Minimization (ERM) with squared loss may attain minimax suboptimal error rates. We show that under mild assumptions, the suboptimality of ERM must be due to large bias rather than variance. We also show that our estimates imply stability of ERM, complementing the main result of Caponnetto and Rakhlin (2006) for non-Donsker classes.
  arXiv  Detail & Related papers  (2023-05-29T15:25:48Z)
• What Is Missing in IRM Training and Evaluation? Challenges and Solutions [41.56612265456626]
  Invariant risk minimization (IRM) has received increasing attention as a way to acquire environment-agnostic data representations and predictions. Recent works have found that the optimality of the originally-proposed IRM optimization (IRM) may be compromised in practice. We identify and resolve three practical limitations in IRM training and evaluation.
  arXiv  Detail & Related papers  (2023-03-04T07:06:24Z)
• A State-Augmented Approach for Learning Optimal Resource Management Decisions in Wireless Networks [58.720142291102135]
  We consider a radio resource management (RRM) problem in a multi-user wireless network. The goal is to optimize a network-wide utility function subject to constraints on the ergodic average performance of users. We propose a state-augmented parameterization for the RRM policy, where alongside the instantaneous network states, the RRM policy takes as input the set of dual variables corresponding to the constraints.
  arXiv  Detail & Related papers  (2022-10-28T21:24:13Z)
• The Missing Invariance Principle Found -- the Reciprocal Twin of Invariant Risk Minimization [7.6146285961466]
  In Risk Minimization (IRM) can fail to generalize poorly to out-of-distribution (OOD) data. We show that MRI-v1 can guarantee invariant predictors given sufficient environments. We also demonstrate that MRI strongly out-performs IRM and achieves a near-optimal OOD in image-based problems.
  arXiv  Detail & Related papers  (2022-05-29T00:14:51Z)
• Effective Mutation Rate Adaptation through Group Elite Selection [50.88204196504888]
  This paper introduces the Group Elite Selection of Mutation Rates (GESMR) algorithm. GESMR co-evolves a population of solutions and a population of MRs, such that each MR is assigned to a group of solutions. With the same number of function evaluations and with almost no overhead, GESMR converges faster and to better solutions than previous approaches.
  arXiv  Detail & Related papers  (2022-04-11T01:08:26Z)
• Meta-Learned Invariant Risk Minimization [12.6484257912092]
  Empirical Risk Minimization (ERM) based machine learning algorithms have suffered from weak generalization performance on data obtained from out-of-distribution (OOD) In this paper, we propose a novel meta-learning based approach for IRM. We show that our algorithm not only has better OOD generalization performance than IRMv1 and all IRM variants, but also addresses the weakness of IRMv1 with improved stability.
  arXiv  Detail & Related papers  (2021-03-24T02:52:48Z)
• The Risks of Invariant Risk Minimization [52.7137956951533]
  Invariant Risk Minimization is an objective based on the idea for learning deep, invariant features of data. We present the first analysis of classification under the IRM objective--as well as these recently proposed alternatives--under a fairly natural and general model. We show that IRM can fail catastrophically unless the test data are sufficiently similar to the training distribution--this is precisely the issue that it was intended to solve.
  arXiv  Detail & Related papers  (2020-10-12T14:54:32Z)
• Tilted Empirical Risk Minimization [26.87656095874882]
  We show that it is possible to flexibly tune the impact of individual losses through a straightforward extension to empirical risk minimization. We show that TERM can increase or decrease the influence of outliers, respectively, to enable fairness or robustness. It can also enable entirely new applications, such as simultaneously addressing outliers and promoting fairness.
  arXiv  Detail & Related papers  (2020-07-02T14:49:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.