Related papers: Adapting to Shifting Correlations with Unlabeled Data Calibration

Adapting to Shifting Correlations with Unlabeled Data Calibration

URL: http://arxiv.org/abs/2409.05996v1
Date: Mon, 9 Sep 2024 18:45:43 GMT
Title: Adapting to Shifting Correlations with Unlabeled Data Calibration
Authors: Minh Nguyen, Alan Q. Wang, Heejong Kim, Mert R. Sabuncu,
Abstract summary: Distribution shifts between sites can seriously degrade model performance since models are prone to exploiting unstable correlations. We propose Generalized Prevalence Adjustment (GPA), a flexible method that adjusts model predictions to the shifting correlations between prediction target and confounders. GPA can infer the interaction between target and confounders in new sites using unlabeled samples from those sites.
Score: 6.84735357291896
License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
Abstract: Distribution shifts between sites can seriously degrade model performance since models are prone to exploiting unstable correlations. Thus, many methods try to find features that are stable across sites and discard unstable features. However, unstable features might have complementary information that, if used appropriately, could increase accuracy. More recent methods try to adapt to unstable features at the new sites to achieve higher accuracy. However, they make unrealistic assumptions or fail to scale to multiple confounding features. We propose Generalized Prevalence Adjustment (GPA for short), a flexible method that adjusts model predictions to the shifting correlations between prediction target and confounders to safely exploit unstable features. GPA can infer the interaction between target and confounders in new sites using unlabeled samples from those sites. We evaluate GPA on several real and synthetic datasets, and show that it outperforms competitive baselines.

Related papers

Robust Learning via Conditional Prevalence Adjustment [7.480241867887245]
Deep learning models might fail catastrophically in unseen sites. We propose a method called CoPA (Conditional Prevalence-Adjustment) for anti-causal tasks.
arXiv Detail & Related papers (2023-10-24T12:13:49Z)
Spuriosity Didn't Kill the Classifier: Using Invariant Predictions to Harness Spurious Features [19.312258609611686]
Stable Feature Boosting (SFB) is an algorithm for learning a predictor that separates stable and conditionally-independent unstable features. We show that SFB can learn anally-optimal predictor without test-domain labels. Empirically, we demonstrate the effectiveness of SFB on real and synthetic data.
arXiv Detail & Related papers (2023-07-19T12:15:06Z)
Unleashing the Power of Graph Data Augmentation on Covariate Distribution Shift [50.98086766507025]
We propose a simple-yet-effective data augmentation strategy, Adversarial Invariant Augmentation (AIA) AIA aims to extrapolate and generate new environments, while concurrently preserving the original stable features during the augmentation process.
arXiv Detail & Related papers (2022-11-05T07:55:55Z)
Domain Adaptation under Missingness Shift [38.650099178537864]
We introduce the problem of Domain Adaptation under Missingness Shift (DAMS) Rates of missing data often depend on record-keeping policies and thus may change across times and locations. In experiments on synthetic and semi-synthetic data, we demonstrate the promise of our methods when assumptions hold.
arXiv Detail & Related papers (2022-11-03T18:49:38Z)
Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
arXiv Detail & Related papers (2022-01-11T23:01:12Z)
Training on Test Data with Bayesian Adaptation for Covariate Shift [96.3250517412545]
Deep neural networks often make inaccurate predictions with unreliable uncertainty estimates. We derive a Bayesian model that provides for a well-defined relationship between unlabeled inputs under distributional shift and model parameters. We show that our method improves both accuracy and uncertainty estimation.
arXiv Detail & Related papers (2021-09-27T01:09:08Z)
Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
arXiv Detail & Related papers (2020-06-26T13:50:19Z)
Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift [81.74795324629712]
We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness. The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
arXiv Detail & Related papers (2020-06-19T05:08:43Z)
Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
arXiv Detail & Related papers (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.