Out of Distribution Generalization in Machine Learning

• URL: http://arxiv.org/abs/2103.02667v1
• Date: Wed, 3 Mar 2021 20:35:19 GMT
• Title: Out of Distribution Generalization in Machine Learning
• Authors: Martin Arjovsky
• Abstract summary: In everyday situations when models are tested in slightly different data than they were trained on, ML algorithms can fail spectacularly. This research attempts to formally define this problem, what sets of assumptions are reasonable to make in our data. Then, we focus on a certain class of out of distribution problems, their assumptions, and introduce simple algorithms that follow from these assumptions.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine learning has achieved tremendous success in a variety of domains in recent years. However, a lot of these success stories have been in places where the training and the testing distributions are extremely similar to each other. In everyday situations when models are tested in slightly different data than they were trained on, ML algorithms can fail spectacularly. This research attempts to formally define this problem, what sets of assumptions are reasonable to make in our data and what kind of guarantees we hope to obtain from them. Then, we focus on a certain class of out of distribution problems, their assumptions, and introduce simple algorithms that follow from these assumptions that are able to provide more reliable generalization. A central topic in the thesis is the strong link between discovering the causal structure of the data, finding features that are reliable (when using them to predict) regardless of their context, and out of distribution generalization.

Related papers

• A Survey of Deep Long-Tail Classification Advancements [1.6233132273470656]
  Many data distributions in the real world are hardly uniform. Instead, skewed and long-tailed distributions of various kinds are commonly observed. This poses an interesting problem for machine learning, where most algorithms assume or work well with uniformly distributed data. The problem is further exacerbated by current state-of-the-art deep learning models requiring large volumes of training data.
  arXiv  Detail & Related papers  (2024-04-24T01:59:02Z)
• Fairness and Accuracy under Domain Generalization [10.661409428935494]
  Concerns have arisen that machine learning algorithms may be biased against certain social groups. Many approaches have been proposed to make ML models fair, but they typically rely on the assumption that data distributions in training and deployment are identical. We study the transfer of both fairness and accuracy under domain generalization where the data at test time may be sampled from never-before-seen domains.
  arXiv  Detail & Related papers  (2023-01-30T23:10:17Z)
• Generalizing in the Real World with Representation Learning [1.3494312389622642]
  Machine learning (ML) formalizes the problem of getting computers to learn from experience as optimization of performance according to some metric(s) This is in contrast to requiring behaviour specified in advance (e.g. by hard-coded rules) In this thesis I cover some of my work towards better understanding deep net generalization, identify several ways assumptions and problem settings fail to generalize to the real world, and propose ways to address those failures in practice.
  arXiv  Detail & Related papers  (2022-10-18T15:11:09Z)
• Principled Knowledge Extrapolation with GANs [92.62635018136476]
  We study counterfactual synthesis from a new perspective of knowledge extrapolation. We show that an adversarial game with a closed-form discriminator can be used to address the knowledge extrapolation problem. Our method enjoys both elegant theoretical guarantees and superior performance in many scenarios.
  arXiv  Detail & Related papers  (2022-05-21T08:39:42Z)
• Self-balanced Learning For Domain Generalization [64.99791119112503]
  Domain generalization aims to learn a prediction model on multi-domain source data such that the model can generalize to a target domain with unknown statistics. Most existing approaches have been developed under the assumption that the source data is well-balanced in terms of both domain and class. We propose a self-balanced domain generalization framework that adaptively learns the weights of losses to alleviate the bias caused by different distributions of the multi-domain source data.
  arXiv  Detail & Related papers  (2021-08-31T03:17:54Z)
• OoD-Bench: Benchmarking and Understanding Out-of-Distribution Generalization Datasets and Algorithms [28.37021464780398]
  We show that existing OoD algorithms that outperform empirical risk minimization on one distribution shift usually have limitations on the other distribution shift. The new benchmark may serve as a strong foothold that can be resorted to by future OoD generalization research.
  arXiv  Detail & Related papers  (2021-06-07T15:34:36Z)
• Improving Uncertainty Calibration via Prior Augmented Data [56.88185136509654]
  Neural networks have proven successful at learning from complex data distributions by acting as universal function approximators. They are often overconfident in their predictions, which leads to inaccurate and miscalibrated probabilistic predictions. We propose a solution by seeking out regions of feature space where the model is unjustifiably overconfident, and conditionally raising the entropy of those predictions towards that of the prior distribution of the labels.
  arXiv  Detail & Related papers  (2021-02-22T07:02:37Z)
• Testing for Typicality with Respect to an Ensemble of Learned Distributions [5.850572971372637]
  One-sample approaches to the goodness-of-fit problem offer significant computational advantages for online testing. The ability to correctly reject anomalous data in this setting hinges on the accuracy of the model of the base distribution. Existing methods for the one-sample goodness-of-fit problem do not account for the fact that a model of the base distribution is learned. We propose training an ensemble of density models, considering data to be anomalous if the data is anomalous with respect to any member of the ensemble.
  arXiv  Detail & Related papers  (2020-11-11T19:47:46Z)
• A Note on High-Probability versus In-Expectation Guarantees of Generalization Bounds in Machine Learning [95.48744259567837]
  Statistical machine learning theory often tries to give generalization guarantees of machine learning models. Statements made about the performance of machine learning models have to take the sampling process into account. We show how one may transform one statement to another.
  arXiv  Detail & Related papers  (2020-10-06T09:41:35Z)
• Learning while Respecting Privacy and Robustness to Distributional Uncertainties and Adversarial Data [66.78671826743884]
  The distributionally robust optimization framework is considered for training a parametric model. The objective is to endow the trained model with robustness against adversarially manipulated input data. Proposed algorithms offer robustness with little overhead.
  arXiv  Detail & Related papers  (2020-07-07T18:25:25Z)
• Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
  We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers. We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model. Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
  arXiv  Detail & Related papers  (2020-06-22T21:12:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.