Towards Out-Of-Distribution Generalization: A Survey
- URL: http://arxiv.org/abs/2108.13624v2
- Date: Thu, 27 Jul 2023 13:13:11 GMT
- Title: Towards Out-Of-Distribution Generalization: A Survey
- Authors: Jiashuo Liu, Zheyan Shen, Yue He, Xingxuan Zhang, Renzhe Xu, Han Yu,
Peng Cui
- Abstract summary: Out-of-Distribution generalization is an emerging topic of machine learning research.
This paper represents the first comprehensive, systematic review of OOD generalization.
- Score: 46.329995334444156
- License: http://creativecommons.org/licenses/by-nc-sa/4.0/
- Abstract: Traditional machine learning paradigms are based on the assumption that both
training and test data follow the same statistical pattern, which is
mathematically referred to as Independent and Identically Distributed
($i.i.d.$). However, in real-world applications, this $i.i.d.$ assumption often
fails to hold due to unforeseen distributional shifts, leading to considerable
degradation in model performance upon deployment. This observed discrepancy
indicates the significance of investigating the Out-of-Distribution (OOD)
generalization problem. OOD generalization is an emerging topic of machine
learning research that focuses on complex scenarios wherein the distributions
of the test data differ from those of the training data. This paper represents
the first comprehensive, systematic review of OOD generalization, encompassing
a spectrum of aspects from problem definition, methodological development, and
evaluation procedures, to the implications and future directions of the field.
Our discussion begins with a precise, formal characterization of the OOD
generalization problem. Following that, we categorize existing methodologies
into three segments: unsupervised representation learning, supervised model
learning, and optimization, according to their positions within the overarching
learning process. We provide an in-depth discussion on representative
methodologies for each category, further elucidating the theoretical links
between them. Subsequently, we outline the prevailing benchmark datasets
employed in OOD generalization studies. To conclude, we overview the existing
body of work in this domain and suggest potential avenues for future research
on OOD generalization. A summary of the OOD generalization methodologies
surveyed in this paper can be accessed at
http://out-of-distribution-generalization.com.
Related papers
- Bridging OOD Detection and Generalization: A Graph-Theoretic View [21.84304334604601]
We introduce a graph-theoretic framework to tackle both OOD generalization and detection problems.
By leveraging the graph formulation, data representations are obtained through the factorization of the graph's adjacency matrix.
Empirical results showcase competitive performance in comparison to existing methods.
arXiv Detail & Related papers (2024-09-26T18:35:51Z) - Revisiting Spurious Correlation in Domain Generalization [12.745076668687748]
We build a structural causal model (SCM) to describe the causality within data generation process.
We further conduct a thorough analysis of the mechanisms underlying spurious correlation.
In this regard, we propose to control confounding bias in OOD generalization by introducing a propensity score weighted estimator.
arXiv Detail & Related papers (2024-06-17T13:22:00Z) - A Survey on Evaluation of Out-of-Distribution Generalization [41.39827887375374]
Out-of-Distribution (OOD) generalization is a complex and fundamental problem.
This paper serves as the first effort to conduct a comprehensive review of OOD evaluation.
We categorize existing research into three paradigms: OOD performance testing, OOD performance prediction, and OOD intrinsic property characterization.
arXiv Detail & Related papers (2024-03-04T09:30:35Z) - Towards out-of-distribution generalization in large-scale astronomical
surveys: robust networks learn similar representations [3.653721769378018]
We use Centered Kernel Alignment (CKA), a similarity measure metric of neural network representations, to examine the relationship between representation similarity and performance.
We find that when models are robust to a distribution shift, they produce substantially different representations across their layers on OOD data.
We discuss the potential application of similarity representation in guiding model design, training strategy, and mitigating the OOD problem by incorporating CKA as an inductive bias during training.
arXiv Detail & Related papers (2023-11-29T19:00:05Z) - DIVERSIFY: A General Framework for Time Series Out-of-distribution
Detection and Generalization [58.704753031608625]
Time series is one of the most challenging modalities in machine learning research.
OOD detection and generalization on time series tend to suffer due to its non-stationary property.
We propose DIVERSIFY, a framework for OOD detection and generalization on dynamic distributions of time series.
arXiv Detail & Related papers (2023-08-04T12:27:11Z) - Graph Structure and Feature Extrapolation for Out-of-Distribution Generalization [54.64375566326931]
Out-of-distribution (OOD) generalization deals with the prevalent learning scenario where test distribution shifts from training distribution.
We propose to achieve graph OOD generalization with the novel design of non-Euclidean-space linear extrapolation.
Our design tailors OOD samples for specific shifts without corrupting underlying causal mechanisms.
arXiv Detail & Related papers (2023-06-13T18:46:28Z) - SimSCOOD: Systematic Analysis of Out-of-Distribution Generalization in
Fine-tuned Source Code Models [58.78043959556283]
We study the behaviors of models under different fine-tuning methodologies, including full fine-tuning and Low-Rank Adaptation (LoRA) fine-tuning methods.
Our analysis uncovers that LoRA fine-tuning consistently exhibits significantly better OOD generalization performance than full fine-tuning across various scenarios.
arXiv Detail & Related papers (2022-10-10T16:07:24Z) - Domain Generalization -- A Causal Perspective [20.630396283221838]
Machine learning models have gained widespread success, from healthcare to personalized recommendations.
One of the preliminary assumptions of these models is the independent and identical distribution.
Since the models rely heavily on this assumption, they exhibit poor generalization capabilities.
arXiv Detail & Related papers (2022-09-30T01:56:49Z) - General Greedy De-bias Learning [163.65789778416172]
We propose a General Greedy De-bias learning framework (GGD), which greedily trains the biased models and the base model like gradient descent in functional space.
GGD can learn a more robust base model under the settings of both task-specific biased models with prior knowledge and self-ensemble biased model without prior knowledge.
arXiv Detail & Related papers (2021-12-20T14:47:32Z) - In Search of Robust Measures of Generalization [79.75709926309703]
We develop bounds on generalization error, optimization error, and excess risk.
When evaluated empirically, most of these bounds are numerically vacuous.
We argue that generalization measures should instead be evaluated within the framework of distributional robustness.
arXiv Detail & Related papers (2020-10-22T17:54:25Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.