A Fine-Grained Analysis on Distribution Shift

• URL: http://arxiv.org/abs/2110.11328v1
• Date: Thu, 21 Oct 2021 17:57:08 GMT
• Title: A Fine-Grained Analysis on Distribution Shift
• Authors: Olivia Wiles and Sven Gowal and Florian Stimberg and Sylvestre Alvise-Rebuffi and Ira Ktena and Krishnamurthy (Dj) Dvijotham and Taylan Cemgil
• Abstract summary: We introduce a framework that enables fine-grained analysis of various distribution shifts. We evaluate 19 distinct methods grouped into five categories across both synthetic and real-world datasets. Our framework can be easily extended to include new methods, shifts, and datasets.
• Score: 24.084676204709723
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Robustness to distribution shifts is critical for deploying machine learning models in the real world. Despite this necessity, there has been little work in defining the underlying mechanisms that cause these shifts and evaluating the robustness of algorithms across multiple, different distribution shifts. To this end, we introduce a framework that enables fine-grained analysis of various distribution shifts. We provide a holistic analysis of current state-of-the-art methods by evaluating 19 distinct methods grouped into five categories across both synthetic and real-world datasets. Overall, we train more than 85K models. Our experimental framework can be easily extended to include new methods, shifts, and datasets. We find, unlike previous work~\citep{Gulrajani20}, that progress has been made over a standard ERM baseline; in particular, pretraining and augmentations (learned or heuristic) offer large gains in many cases. However, the best methods are not consistent over different datasets and shifts.

Related papers

• Distributionally Robust Safe Sample Elimination under Covariate Shift [16.85444622474742]
  We consider a machine learning setup where one training dataset is used to train multiple models across slightly different data distributions. We propose the DRSSS method, which combines distributionally robust (DR) optimization and safe sample screening (SSS) The key benefit of this method is that models trained on the reduced dataset will perform the same as those trained on the full dataset for all possible different environments.
  arXiv  Detail & Related papers  (2024-06-10T01:46:42Z)
• Multiply Robust Estimation for Local Distribution Shifts with Multiple Domains [9.429772474335122]
  We focus on scenarios where data distributions vary across multiple segments of the entire population. We propose a two-stage multiply robust estimation method to improve model performance on each individual segment. Our method is designed to be implemented with commonly used off-the-shelf machine learning models.
  arXiv  Detail & Related papers  (2024-02-21T22:01:10Z)
• Tackling Computational Heterogeneity in FL: A Few Theoretical Insights [68.8204255655161]
  We introduce and analyse a novel aggregation framework that allows for formalizing and tackling computational heterogeneous data. Proposed aggregation algorithms are extensively analyzed from a theoretical, and an experimental prospective.
  arXiv  Detail & Related papers  (2023-07-12T16:28:21Z)
• Rethinking Distribution Shifts: Empirical Analysis and Inductive Modeling for Tabular Data [30.518020409197767]
  We build an empirical testbed comprising natural shifts across 5 datasets and 60,000 method configurations. We find $Y|X$-shifts are most prevalent on our testbed, in stark contrast to the heavy focus on $X$ (co)-shifts in the ML literature.
  arXiv  Detail & Related papers  (2023-07-11T14:25:10Z)
• Change is Hard: A Closer Look at Subpopulation Shift [48.0369745740936]
  We propose a unified framework that dissects and explains common shifts in subgroups. We then establish a benchmark of 20 state-of-the-art algorithms evaluated on 12 real-world datasets in vision, language, and healthcare domains.
  arXiv  Detail & Related papers  (2023-02-23T18:59:56Z)
• An Empirical Study on Distribution Shift Robustness From the Perspective of Pre-Training and Data Augmentation [91.62129090006745]
  This paper studies the distribution shift problem from the perspective of pre-training and data augmentation. We provide the first comprehensive empirical study focusing on pre-training and data augmentation.
  arXiv  Detail & Related papers  (2022-05-25T13:04:53Z)
• Structurally Diverse Sampling Reduces Spurious Correlations in Semantic Parsing Datasets [51.095144091781734]
  We propose a novel algorithm for sampling a structurally diverse set of instances from a labeled instance pool with structured outputs. We show that our algorithm performs competitively with or better than prior algorithms in not only compositional template splits but also traditional IID splits. In general, we find that diverse train sets lead to better generalization than random training sets of the same size in 9 out of 10 dataset-split pairs.
  arXiv  Detail & Related papers  (2022-03-16T07:41:27Z)
• Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning [96.75889543560497]
  In many real-world problems, collecting a large number of labeled samples is infeasible. Few-shot learning is the dominant approach to address this issue, where the objective is to quickly adapt to novel categories in presence of a limited number of samples. We propose a novel training mechanism that simultaneously enforces equivariance and invariance to a general set of geometric transformations.
  arXiv  Detail & Related papers  (2021-03-01T21:14:33Z)
• WILDS: A Benchmark of in-the-Wild Distribution Shifts [157.53410583509924]
  Distribution shifts can substantially degrade the accuracy of machine learning systems deployed in the wild. We present WILDS, a curated collection of 8 benchmark datasets that reflect a diverse range of distribution shifts. We show that standard training results in substantially lower out-of-distribution than in-distribution performance.
  arXiv  Detail & Related papers  (2020-12-14T11:14:56Z)
• The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalization [64.61630743818024]
  We introduce four new real-world distribution shift datasets consisting of changes in image style, image blurriness, geographic location, camera operation, and more. We find that using larger models and artificial data augmentations can improve robustness on real-world distribution shifts, contrary to claims in prior work. We also introduce a new data augmentation method which advances the state-of-the-art and outperforms models pretrained with 1000 times more labeled data.
  arXiv  Detail & Related papers  (2020-06-29T17:59:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.