Importance Weighting Correction of Regularized Least-Squares for Covariate and Target Shifts

• URL: http://arxiv.org/abs/2210.09709v1
• Date: Tue, 18 Oct 2022 09:39:36 GMT
• Title: Importance Weighting Correction of Regularized Least-Squares for Covariate and Target Shifts
• Authors: Davit Gogolashvili
• Abstract summary: In many real world problems, the training data and test data have different distributions. Importance weighting (IW) correction is a universal method for correcting the bias present in learning scenarios under dataset shift. We show that IW correction works equally well for different dataset shift scenarios.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In many real world problems, the training data and test data have different distributions. This situation is commonly referred as a dataset shift. The most common settings for dataset shift often considered in the literature are {\em covariate shift } and {\em target shift}. Importance weighting (IW) correction is a universal method for correcting the bias present in learning scenarios under dataset shift. The question one may ask is: does IW correction work equally well for different dataset shift scenarios? By investigating the generalization properties of the weighted kernel ridge regression (W-KRR) under covariate and target shifts we show that the answer is negative, except when IW is bounded and the model is wellspecified. In the latter cases, a minimax optimal rates are achieved by importance weighted kernel ridge regression (IW-KRR) in both, covariate and target shift scenarios. Slightly relaxing the boundedness condition of the IW we show that the IW-KRR still achieves the optimal rates under target shift while leading to slower rates for covariate shift. In the case of the model misspecification we show that the performance of the W-KRR under covariate shift could be substantially increased by designing an alternative reweighting function. The distinction between misspecified and wellspecified scenarios does not seem to be crucial in the learning problems under target shift.

Related papers

• When is Importance Weighting Correction Needed for Covariate Shift Adaptation? [15.622171482618805]
  This paper investigates when the importance weighting (IW) correction is needed in supervised learning. Classic results show that the IW correction is needed when the model is parametric and misspecified. Recent results indicate that the IW correction may not be necessary when the model is nonparametric and well-specified.
  arXiv  Detail & Related papers  (2023-03-07T16:37:30Z)
• Adapting to Latent Subgroup Shifts via Concepts and Proxies [82.01141290360562]
  We show that the optimal target predictor can be non-parametrically identified with the help of concept and proxy variables available only in the source domain. For continuous observations, we propose a latent variable model specific to the data generation process at hand.
  arXiv  Detail & Related papers  (2022-12-21T18:30:22Z)
• Unleashing the Power of Graph Data Augmentation on Covariate Distribution Shift [50.98086766507025]
  We propose a simple-yet-effective data augmentation strategy, Adversarial Invariant Augmentation (AIA) AIA aims to extrapolate and generate new environments, while concurrently preserving the original stable features during the augmentation process.
  arXiv  Detail & Related papers  (2022-11-05T07:55:55Z)
• The Power and Limitation of Pretraining-Finetuning for Linear Regression under Covariate Shift [127.21287240963859]
  We investigate a transfer learning approach with pretraining on the source data and finetuning based on the target data. For a large class of linear regression instances, transfer learning with $O(N2)$ source data is as effective as supervised learning with $N$ target data.
  arXiv  Detail & Related papers  (2022-08-03T05:59:49Z)
• Towards Backwards-Compatible Data with Confounded Domain Adaptation [0.0]
  We seek to achieve general-purpose data backwards compatibility by modifying generalized label shift (GLS) We present a novel framework for this problem, based on minimizing the expected divergence between the source and target conditional distributions. We provide concrete implementations using the Gaussian reverse Kullback-Leibler divergence and the maximum mean discrepancy.
  arXiv  Detail & Related papers  (2022-03-23T20:53:55Z)
• PAC Prediction Sets Under Covariate Shift [40.67733209235852]
  Uncertainty is important when there are changes to the underlying data distribution. Most existing uncertainty quantification algorithms break down in the presence of such shifts. We propose a novel approach that addresses this challenge by constructing emphprobably approximately correct (PAC) prediction sets.
  arXiv  Detail & Related papers  (2021-06-17T23:28:42Z)
• KL Guided Domain Adaptation [88.19298405363452]
  Domain adaptation is an important problem and often needed for real-world applications. A common approach in the domain adaptation literature is to learn a representation of the input that has the same distributions over the source and the target domain. We show that with a probabilistic representation network, the KL term can be estimated efficiently via minibatch samples.
  arXiv  Detail & Related papers  (2021-06-14T22:24:23Z)
• Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correction [49.25830718574892]
  We present a new framework named Tail-to-Tail (textbfTtT) non-autoregressive sequence prediction. Considering that most tokens are correct and can be conveyed directly from source to target, and the error positions can be estimated and corrected. Experimental results on standard datasets, especially on the variable-length datasets, demonstrate the effectiveness of TtT in terms of sentence-level Accuracy, Precision, Recall, and F1-Measure.
  arXiv  Detail & Related papers  (2021-06-03T05:56:57Z)
• Evaluating Prediction-Time Batch Normalization for Robustness under Covariate Shift [81.74795324629712]
  We call prediction-time batch normalization, which significantly improves model accuracy and calibration under covariate shift. We show that prediction-time batch normalization provides complementary benefits to existing state-of-the-art approaches for improving robustness. The method has mixed results when used alongside pre-training, and does not seem to perform as well under more natural types of dataset shift.
  arXiv  Detail & Related papers  (2020-06-19T05:08:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.