When is Importance Weighting Correction Needed for Covariate Shift Adaptation?

• URL: http://arxiv.org/abs/2303.04020v1
• Date: Tue, 7 Mar 2023 16:37:30 GMT
• Title: When is Importance Weighting Correction Needed for Covariate Shift Adaptation?
• Authors: Davit Gogolashvili, Matteo Zecchin, Motonobu Kanagawa, Marios Kountouris, Maurizio Filippone
• Abstract summary: This paper investigates when the importance weighting (IW) correction is needed in supervised learning. Classic results show that the IW correction is needed when the model is parametric and misspecified. Recent results indicate that the IW correction may not be necessary when the model is nonparametric and well-specified.
• Score: 15.622171482618805
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper investigates when the importance weighting (IW) correction is needed to address covariate shift, a common situation in supervised learning where the input distributions of training and test data differ. Classic results show that the IW correction is needed when the model is parametric and misspecified. In contrast, recent results indicate that the IW correction may not be necessary when the model is nonparametric and well-specified. We examine the missing case in the literature where the model is nonparametric and misspecified, and show that the IW correction is needed for obtaining the best approximation of the true unknown function for the test distribution. We do this by analyzing IW-corrected kernel ridge regression, covering a variety of settings, including parametric and nonparametric models, well-specified and misspecified settings, and arbitrary weighting functions.

Related papers

• Normalizing Flows for Conformal Regression [0.0]
  Conformal Prediction (CP) algorithms estimate the uncertainty of a prediction model by calibrating its outputs on labeled data. We present a general scheme to localize the intervals by training the calibration process. Unlike the Error Reweighting CP algorithm of Papadopoulos et al. (2008), the framework allows estimating the gap between nominal and empirical conditional validity.
  arXiv  Detail & Related papers  (2024-06-05T15:04:28Z)
• Schroedinger's Threshold: When the AUC doesn't predict Accuracy [6.091702876917282]
  Area Under Curve measure (AUC) seems apt to evaluate and compare diverse models. We show that the AUC yields an academic and optimistic notion of accuracy that can misalign with the actual accuracy observed in application.
  arXiv  Detail & Related papers  (2024-04-04T10:18:03Z)
• Importance Weighting Correction of Regularized Least-Squares for Covariate and Target Shifts [0.0]
  In many real world problems, the training data and test data have different distributions. Importance weighting (IW) correction is a universal method for correcting the bias present in learning scenarios under dataset shift. We show that IW correction works equally well for different dataset shift scenarios.
  arXiv  Detail & Related papers  (2022-10-18T09:39:36Z)
• Parametric and Multivariate Uncertainty Calibration for Regression and Object Detection [4.630093015127541]
  We show that common detection models overestimate the spatial uncertainty in comparison to the observed error. Our experiments show that the simple Isotonic Regression recalibration method is sufficient to achieve a good calibrated uncertainty. In contrast, if normal distributions are required for subsequent processes, our GP-Normal recalibration method yields the best results.
  arXiv  Detail & Related papers  (2022-07-04T08:00:20Z)
• T-Cal: An optimal test for the calibration of predictive models [49.11538724574202]
  We consider detecting mis-calibration of predictive models using a finite validation dataset as a hypothesis testing problem. detecting mis-calibration is only possible when the conditional probabilities of the classes are sufficiently smooth functions of the predictions. We propose T-Cal, a minimax test for calibration based on a de-biased plug-in estimator of the $ell$-Expected Error (ECE)
  arXiv  Detail & Related papers  (2022-03-03T16:58:54Z)
• Training on Test Data with Bayesian Adaptation for Covariate Shift [96.3250517412545]
  Deep neural networks often make inaccurate predictions with unreliable uncertainty estimates. We derive a Bayesian model that provides for a well-defined relationship between unlabeled inputs under distributional shift and model parameters. We show that our method improves both accuracy and uncertainty estimation.
  arXiv  Detail & Related papers  (2021-09-27T01:09:08Z)
• Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correction [49.25830718574892]
  We present a new framework named Tail-to-Tail (textbfTtT) non-autoregressive sequence prediction. Considering that most tokens are correct and can be conveyed directly from source to target, and the error positions can be estimated and corrected. Experimental results on standard datasets, especially on the variable-length datasets, demonstrate the effectiveness of TtT in terms of sentence-level Accuracy, Precision, Recall, and F1-Measure.
  arXiv  Detail & Related papers  (2021-06-03T05:56:57Z)
• Unsupervised Calibration under Covariate Shift [92.02278658443166]
  We introduce the problem of calibration under domain shift and propose an importance sampling based approach to address it. We evaluate and discuss the efficacy of our method on both real-world datasets and synthetic datasets.
  arXiv  Detail & Related papers  (2020-06-29T21:50:07Z)
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift [81.74795324629712]
  We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness. The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
  arXiv  Detail & Related papers  (2020-06-19T05:08:43Z)
• On the Inference Calibration of Neural Machine Translation [54.48932804996506]
  We study the correlation between calibration and translation performance and linguistic properties of miscalibration. We propose a new graduated label smoothing method that can improve both inference calibration and translation performance.
  arXiv  Detail & Related papers  (2020-05-03T02:03:56Z)
• Stable Prediction with Model Misspecification and Agnostic Distribution Shift [41.26323389341987]
  In machine learning algorithms, two main assumptions are required to guarantee performance. One is that the test data are drawn from the same distribution as the training data, and the other is that the model is correctly specified. Under model misspecification, distribution shift between training and test data leads to inaccuracy of parameter estimation and instability of prediction across unknown test data. We propose a novel Decorrelated Weighting Regression (DWR) algorithm which jointly optimize a variable decorrelation regularizer and a weighted regression model.
  arXiv  Detail & Related papers  (2020-01-31T08:56:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.