Towards out of distribution generalization for problems in mechanics

• URL: http://arxiv.org/abs/2206.14917v1
• Date: Wed, 29 Jun 2022 21:14:08 GMT
• Title: Towards out of distribution generalization for problems in mechanics
• Authors: Lingxiao Yuan, Harold S. Park, Emma Lejeune
• Abstract summary: Out-of-distribution (OOD) generalization assumes that the test data may shift. Traditional machine learning (ML) methods rely on the assumption that the training (observed) data and testing (unseen) data are independent and identically distributed.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: There has been a massive increase in research interest towards applying data driven methods to problems in mechanics. While traditional machine learning (ML) methods have enabled many breakthroughs, they rely on the assumption that the training (observed) data and testing (unseen) data are independent and identically distributed (i.i.d). Thus, traditional ML approaches often break down when applied to real world mechanics problems with unknown test environments and data distribution shifts. In contrast, out-of-distribution (OOD) generalization assumes that the test data may shift (i.e., violate the i.i.d. assumption). To date, multiple methods have been proposed to improve the OOD generalization of ML methods. However, because of the lack of benchmark datasets for OOD regression problems, the efficiency of these OOD methods on regression problems, which dominate the mechanics field, remains unknown. To address this, we investigate the performance of OOD generalization methods for regression problems in mechanics. Specifically, we identify three OOD problems: covariate shift, mechanism shift, and sampling bias. For each problem, we create two benchmark examples that extend the Mechanical MNIST dataset collection, and we investigate the performance of popular OOD generalization methods on these mechanics-specific regression problems. Our numerical experiments show that in most cases, while the OOD generalization algorithms perform better compared to traditional ML methods on these OOD problems, there is a compelling need to develop more robust OOD generalization methods that are effective across multiple OOD scenarios. Overall, we expect that this study, as well as the associated open access benchmark datasets, will enable further development of OOD generalization methods for mechanics specific regression problems.

Related papers

• Mixture Data for Training Cannot Ensure Out-of-distribution Generalization [21.801115344132114]
  We show that increasing the size of training data does not always lead to a reduction in the test generalization error. In this work, we quantitatively redefine OOD data as those situated outside the convex hull of mixed training data. Our proof of the new risk bound agrees that the efficacy of well-trained models can be guaranteed for unseen data.
  arXiv  Detail & Related papers  (2023-12-25T11:00:38Z)
• Graph Structure and Feature Extrapolation for Out-of-Distribution Generalization [54.64375566326931]
  Out-of-distribution (OOD) generalization deals with the prevalent learning scenario where test distribution shifts from training distribution. We propose to achieve graph OOD generalization with the novel design of non-Euclidean-space linear extrapolation. Our design tailors OOD samples for specific shifts without corrupting underlying causal mechanisms.
  arXiv  Detail & Related papers  (2023-06-13T18:46:28Z)
• Pseudo-OOD training for robust language models [78.15712542481859]
  OOD detection is a key component of a reliable machine-learning model for any industry-scale application. We propose POORE - POsthoc pseudo-Ood REgularization, that generates pseudo-OOD samples using in-distribution (IND) data. We extensively evaluate our framework on three real-world dialogue systems, achieving new state-of-the-art in OOD detection.
  arXiv  Detail & Related papers  (2022-10-17T14:32:02Z)
• Improved OOD Generalization via Conditional Invariant Regularizer [43.62211060412388]
  We show that given a class label, conditionally independent models of spurious attributes are OOD general. Based on this, metric Conditional Variation (CSV) which controls OOD error is proposed to measure such conditional independence. An algorithm with minicave convergence rate is proposed to solve the problem.
  arXiv  Detail & Related papers  (2022-07-14T06:34:21Z)
• Rethinking Machine Learning Robustness via its Link with the Out-of-Distribution Problem [16.154434566725012]
  We investigate the causes behind machine learning models' susceptibility to adversarial examples. We propose an OOD generalization method that stands against both adversary-induced and natural distribution shifts. Our approach consistently improves robustness to OOD adversarial inputs and outperforms state-of-the-art defenses.
  arXiv  Detail & Related papers  (2022-02-18T00:17:23Z)
• Training OOD Detectors in their Natural Habitats [31.565635192716712]
  Out-of-distribution (OOD) detection is important for machine learning models deployed in the wild. Recent methods use auxiliary outlier data to regularize the model for improved OOD detection. We propose a novel framework that leverages wild mixture data -- that naturally consists of both ID and OOD samples.
  arXiv  Detail & Related papers  (2022-02-07T15:38:39Z)
• No True State-of-the-Art? OOD Detection Methods are Inconsistent across Datasets [69.725266027309]
  Out-of-distribution detection is an important component of reliable ML systems. In this work, we show that none of these methods are inherently better at OOD detection than others on a standardized set of 16 pairs. We also show that a method outperforming another on a certain (ID, OOD) pair may not do so in a low-data regime.
  arXiv  Detail & Related papers  (2021-09-12T16:35:00Z)
• Towards Out-Of-Distribution Generalization: A Survey [46.329995334444156]
  Out-of-Distribution generalization is an emerging topic of machine learning research. This paper represents the first comprehensive, systematic review of OOD generalization.
  arXiv  Detail & Related papers  (2021-08-31T05:28:42Z)
• Improved OOD Generalization via Adversarial Training and Pre-training [49.08683910076778]
  In this paper, we theoretically show that a model robust to input perturbations generalizes well on OOD data. Inspired by previous findings that adversarial training helps improve input-robustness, we show that adversarially trained models have converged excess risk on OOD data.
  arXiv  Detail & Related papers  (2021-05-24T08:06:35Z)
• Learn what you can't learn: Regularized Ensembles for Transductive Out-of-distribution Detection [76.39067237772286]
  We show that current out-of-distribution (OOD) detection algorithms for neural networks produce unsatisfactory results in a variety of OOD detection scenarios. This paper studies how such "hard" OOD scenarios can benefit from adjusting the detection method after observing a batch of the test data. We propose a novel method that uses an artificial labeling scheme for the test data and regularization to obtain ensembles of models that produce contradictory predictions only on the OOD samples in a test batch.
  arXiv  Detail & Related papers  (2020-12-10T16:55:13Z)
• On the Value of Out-of-Distribution Testing: An Example of Goodhart's Law [78.10523907729642]
  VQA-CP has become the standard OOD benchmark for visual question answering. Most published methods rely on explicit knowledge of the construction of the OOD splits. We show that embarrassingly-simple methods, including one that generates answers at random, surpass the state of the art on some question types.
  arXiv  Detail & Related papers  (2020-05-19T06:45:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.