Class Is Invariant to Context and Vice Versa: On Learning Invariance for
Out-Of-Distribution Generalization
- URL: http://arxiv.org/abs/2208.03462v2
- Date: Fri, 31 Mar 2023 05:56:37 GMT
- Title: Class Is Invariant to Context and Vice Versa: On Learning Invariance for
Out-Of-Distribution Generalization
- Authors: Jiaxin Qi, Kaihua Tang, Qianru Sun, Xian-Sheng Hua, and Hanwang Zhang
- Abstract summary: We argue that the widely adopted assumption in prior work, the context bias can be directly annotated or estimated from biased class prediction.
In contrast, we point out the everoverlooked other side of the above principle: context is also invariant to class.
We implement this idea by minimizing the contrastive loss of intra-class sample similarity while assuring this similarity to be invariant across all classes.
- Score: 80.22736530704191
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Out-Of-Distribution generalization (OOD) is all about learning invariance
against environmental changes. If the context in every class is evenly
distributed, OOD would be trivial because the context can be easily removed due
to an underlying principle: class is invariant to context. However, collecting
such a balanced dataset is impractical. Learning on imbalanced data makes the
model bias to context and thus hurts OOD. Therefore, the key to OOD is context
balance. We argue that the widely adopted assumption in prior work, the context
bias can be directly annotated or estimated from biased class prediction,
renders the context incomplete or even incorrect. In contrast, we point out the
everoverlooked other side of the above principle: context is also invariant to
class, which motivates us to consider the classes (which are already labeled)
as the varying environments to resolve context bias (without context labels).
We implement this idea by minimizing the contrastive loss of intra-class sample
similarity while assuring this similarity to be invariant across all classes.
On benchmarks with various context biases and domain gaps, we show that a
simple re-weighting based classifier equipped with our context estimation
achieves state-of-the-art performance. We provide the theoretical
justifications in Appendix and codes on
https://github.com/simpleshinobu/IRMCon.
Related papers
- Effect-Invariant Mechanisms for Policy Generalization [3.701112941066256]
It has been suggested to exploit invariant conditional distributions to learn models that generalize better to unseen environments.
We introduce a relaxation of full invariance called effect-invariance and prove that it is sufficient, under suitable assumptions, for zero-shot policy generalization.
We present empirical results using simulated data and a mobile health intervention dataset to demonstrate the effectiveness of our approach.
arXiv Detail & Related papers (2023-06-19T14:50:24Z) - Malign Overfitting: Interpolation Can Provably Preclude Invariance [30.776243638012314]
We show that "benign overfitting" in which models generalize well despite interpolating might not favorably extend to settings in which robustness or fairness are desirable.
We propose and analyze an algorithm that successfully learns a non-interpolating classifier that is provably invariant.
arXiv Detail & Related papers (2022-11-28T19:17:31Z) - In What Ways Are Deep Neural Networks Invariant and How Should We
Measure This? [5.757836174655293]
We introduce a family of invariance and equivariance metrics that allows us to quantify these properties in a way that disentangles them from other metrics such as loss or accuracy.
We draw a range of conclusions about invariance and equivariance in deep learning models, ranging from whether initializing a model with pretrained weights has an effect on a trained model's invariance, to the extent to which invariance learned via training can generalize to out-of-distribution data.
arXiv Detail & Related papers (2022-10-07T18:43:21Z) - Equivariance and Invariance Inductive Bias for Learning from
Insufficient Data [65.42329520528223]
We show why insufficient data renders the model more easily biased to the limited training environments that are usually different from testing.
We propose a class-wise invariant risk minimization (IRM) that efficiently tackles the challenge of missing environmental annotation in conventional IRM.
arXiv Detail & Related papers (2022-07-25T15:26:19Z) - Predicting Out-of-Domain Generalization with Neighborhood Invariance [59.05399533508682]
We propose a measure of a classifier's output invariance in a local transformation neighborhood.
Our measure is simple to calculate, does not depend on the test point's true label, and can be applied even in out-of-domain (OOD) settings.
In experiments on benchmarks in image classification, sentiment analysis, and natural language inference, we demonstrate a strong and robust correlation between our measure and actual OOD generalization.
arXiv Detail & Related papers (2022-07-05T14:55:16Z) - Do Deep Networks Transfer Invariances Across Classes? [123.84237389985236]
We show how a generative approach for learning the nuisance transformations can help transfer invariances across classes.
Our results provide one explanation for why classifiers generalize poorly on unbalanced and longtailed distributions.
arXiv Detail & Related papers (2022-03-18T04:38:18Z) - Unsupervised Feature Learning by Cross-Level Instance-Group
Discrimination [68.83098015578874]
We integrate between-instance similarity into contrastive learning, not directly by instance grouping, but by cross-level discrimination.
CLD effectively brings unsupervised learning closer to natural data and real-world applications.
New state-of-the-art on self-supervision, semi-supervision, and transfer learning benchmarks, and beats MoCo v2 and SimCLR on every reported performance.
arXiv Detail & Related papers (2020-08-09T21:13:13Z) - On Model Evaluation under Non-constant Class Imbalance [0.0]
Many real-world classification problems are significantly class-imbalanced to detriment of the class of interest.
The usual assumption is that the test dataset imbalance equals the real-world imbalance.
We introduce methods focusing on evaluation under non-constant class imbalance.
arXiv Detail & Related papers (2020-01-15T21:52:24Z) - Don't Judge an Object by Its Context: Learning to Overcome Contextual
Bias [113.44471186752018]
Existing models often leverage co-occurrences between objects and their context to improve recognition accuracy.
This work focuses on addressing such contextual biases to improve the robustness of the learnt feature representations.
arXiv Detail & Related papers (2020-01-09T18:31:55Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.