Multiply Robust Estimation for Local Distribution Shifts with Multiple Domains

• URL: http://arxiv.org/abs/2402.14145v2
• Date: Tue, 4 Jun 2024 01:30:07 GMT
• Title: Multiply Robust Estimation for Local Distribution Shifts with Multiple Domains
• Authors: Steven Wilkins-Reeves, Xu Chen, Qi Ma, Christine Agarwal, Aude Hofleitner,
• Abstract summary: We focus on scenarios where data distributions vary across multiple segments of the entire population. We propose a two-stage multiply robust estimation method to improve model performance on each individual segment. Our method is designed to be implemented with commonly used off-the-shelf machine learning models.
• Score: 9.429772474335122
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Distribution shifts are ubiquitous in real-world machine learning applications, posing a challenge to the generalization of models trained on one data distribution to another. We focus on scenarios where data distributions vary across multiple segments of the entire population and only make local assumptions about the differences between training and test (deployment) distributions within each segment. We propose a two-stage multiply robust estimation method to improve model performance on each individual segment for tabular data analysis. The method involves fitting a linear combination of the based models, learned using clusters of training data from multiple segments, followed by a refinement step for each segment. Our method is designed to be implemented with commonly used off-the-shelf machine learning models. We establish theoretical guarantees on the generalization bound of the method on the test risk. With extensive experiments on synthetic and real datasets, we demonstrate that the proposed method substantially improves over existing alternatives in prediction accuracy and robustness on both regression and classification tasks. We also assess its effectiveness on a user city prediction dataset from Meta.

Related papers

• Distributionally Robust Safe Sample Elimination under Covariate Shift [16.85444622474742]
  We consider a machine learning setup where one training dataset is used to train multiple models across slightly different data distributions. We propose the DRSSS method, which combines distributionally robust (DR) optimization and safe sample screening (SSS) The key benefit of this method is that models trained on the reduced dataset will perform the same as those trained on the full dataset for all possible different environments.
  arXiv  Detail & Related papers  (2024-06-10T01:46:42Z)
• Collaborative Heterogeneous Causal Inference Beyond Meta-analysis [68.4474531911361]
  We propose a collaborative inverse propensity score estimator for causal inference with heterogeneous data. Our method shows significant improvements over the methods based on meta-analysis when heterogeneity increases.
  arXiv  Detail & Related papers  (2024-04-24T09:04:36Z)
• Consistency Regularization for Generalizable Source-free Domain Adaptation [62.654883736925456]
  Source-free domain adaptation (SFDA) aims to adapt a well-trained source model to an unlabelled target domain without accessing the source dataset. Existing SFDA methods ONLY assess their adapted models on the target training set, neglecting the data from unseen but identically distributed testing sets. We propose a consistency regularization framework to develop a more generalizable SFDA method.
  arXiv  Detail & Related papers  (2023-08-03T07:45:53Z)
• An Information-Theoretic Approach for Estimating Scenario Generalization in Crowd Motion Prediction [27.10815774845461]
  We propose a novel scoring method, which characterizes generalization of models trained on source crowd scenarios and applied to target crowd scenarios. The Interaction component aims to characterize the difficulty of scenario domains, while the diversity of a scenario domain is captured in the Diversity score. Our experimental results validate the efficacy of the proposed method on several simulated and real-world (source,target) generalization tasks.
  arXiv  Detail & Related papers  (2022-11-02T01:39:30Z)
• Decoupling Predictions in Distributed Learning for Multi-Center Left Atrial MRI Segmentation [20.20518948616193]
  We propose a new framework of distributed learning that bridges the gap between two groups, and improves the performance for both generic and local data. Results on multi-center left atrial (LA) MRI segmentation showed that our method demonstrated superior performance over existing methods on both generic and local data.
  arXiv  Detail & Related papers  (2022-06-10T08:35:42Z)
• DRFLM: Distributionally Robust Federated Learning with Inter-client Noise via Local Mixup [58.894901088797376]
  federated learning has emerged as a promising approach for training a global model using data from multiple organizations without leaking their raw data. We propose a general framework to solve the above two challenges simultaneously. We provide comprehensive theoretical analysis including robustness analysis, convergence analysis, and generalization ability.
  arXiv  Detail & Related papers  (2022-04-16T08:08:29Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)
• Deep Learning with Multiple Data Set: A Weighted Goal Programming Approach [2.7393821783237184]
  Large-scale data analysis is growing at an exponential rate as data proliferates in our societies. Deep Learning models require plenty of resources, and distributed training is needed. This paper presents a Multicriteria approach for distributed learning.
  arXiv  Detail & Related papers  (2021-11-27T07:10:25Z)
• Optimal Ensemble Construction for Multi-Study Prediction with Applications to COVID-19 Excess Mortality Estimation [7.02598981483736]
  Multi-study ensembling uses a two-stage strategy which fits study-specific models and estimates ensemble weights separately. This approach ignores the ensemble properties at the model-fitting stage, potentially resulting in a loss of efficiency. We show that when little data is available for a country before the onset of the pandemic, leveraging data from other countries can substantially improve prediction accuracy.
  arXiv  Detail & Related papers  (2021-09-19T16:52:41Z)
• BREEDS: Benchmarks for Subpopulation Shift [98.90314444545204]
  We develop a methodology for assessing the robustness of models to subpopulation shift. We leverage the class structure underlying existing datasets to control the data subpopulations that comprise the training and test distributions. Applying this methodology to the ImageNet dataset, we create a suite of subpopulation shift benchmarks of varying granularity.
  arXiv  Detail & Related papers  (2020-08-11T17:04:47Z)
• Rethinking Class-Balanced Methods for Long-Tailed Visual Recognition from a Domain Adaptation Perspective [98.70226503904402]
  Object frequency in the real world often follows a power law, leading to a mismatch between datasets with long-tailed class distributions. We propose to augment the classic class-balanced learning by explicitly estimating the differences between the class-conditioned distributions with a meta-learning approach.
  arXiv  Detail & Related papers  (2020-03-24T11:28:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.