Double Debiased Covariate Shift Adaptation Robust to Density-Ratio Estimation

• URL: http://arxiv.org/abs/2310.16638v3
• Date: Sat, 26 Oct 2024 19:19:19 GMT
• Title: Double Debiased Covariate Shift Adaptation Robust to Density-Ratio Estimation
• Authors: Masahiro Kato, Kota Matsui, Ryo Inokuchi,
• Abstract summary: We propose a doubly robust estimator for covariate shift adaptation via importance weighting. Our estimator reduces the bias arising from the density ratio estimation errors. Notably, our estimator remains consistent if either the density ratio estimator or the regression function is consistent.
• Score: 7.8856737627153874
• License:
• Abstract: Consider a scenario where we have access to train data with both covariates and outcomes while test data only contains covariates. In this scenario, our primary aim is to predict the missing outcomes of the test data. With this objective in mind, we train parametric regression models under a covariate shift, where covariate distributions are different between the train and test data. For this problem, existing studies have proposed covariate shift adaptation via importance weighting using the density ratio. This approach averages the train data losses, each weighted by an estimated ratio of the covariate densities between the train and test data, to approximate the test-data risk. Although it allows us to obtain a test-data risk minimizer, its performance heavily relies on the accuracy of the density ratio estimation. Moreover, even if the density ratio can be consistently estimated, the estimation errors of the density ratio also yield bias in the estimators of the regression model's parameters of interest. To mitigate these challenges, we introduce a doubly robust estimator for covariate shift adaptation via importance weighting, which incorporates an additional estimator for the regression function. Leveraging double machine learning techniques, our estimator reduces the bias arising from the density ratio estimation errors. We demonstrate the asymptotic distribution of the regression parameter estimator. Notably, our estimator remains consistent if either the density ratio estimator or the regression function is consistent, showcasing its robustness against potential errors in density ratio estimation. Finally, we confirm the soundness of our proposed method via simulation studies.

Related papers

• Risk and cross validation in ridge regression with correlated samples [72.59731158970894]
  We provide training examples for the in- and out-of-sample risks of ridge regression when the data points have arbitrary correlations. We further extend our analysis to the case where the test point has non-trivial correlations with the training set, setting often encountered in time series forecasting. We validate our theory across a variety of high dimensional data.
  arXiv  Detail & Related papers  (2024-08-08T17:27:29Z)
• Multivariate root-n-consistent smoothing parameter free matching estimators and estimators of inverse density weighted expectations [51.000851088730684]
  We develop novel modifications of nearest-neighbor and matching estimators which converge at the parametric $sqrt n $-rate. We stress that our estimators do not involve nonparametric function estimators and in particular do not rely on sample-size dependent parameters smoothing.
  arXiv  Detail & Related papers  (2024-07-11T13:28:34Z)
• High-dimensional analysis of ridge regression for non-identically distributed data with a variance profile [0.0]
  We study the predictive risk of the ridge estimator for linear regression with a variance profile. For certain class of variance profile, our work highlights the emergence of the well-known double descent phenomenon. We also investigate the similarities and differences that exist with the standard setting of independent and identically distributed data.
  arXiv  Detail & Related papers  (2024-03-29T14:24:49Z)
• Nearest Neighbor Sampling for Covariate Shift Adaptation [7.940293148084844]
  We propose a new covariate shift adaptation method which avoids estimating the weights. The basic idea is to directly work on unlabeled target data, labeled according to the $k$-nearest neighbors in the source dataset. Our experiments show that it achieves drastic reduction in the running time with remarkable accuracy.
  arXiv  Detail & Related papers  (2023-12-15T17:28:09Z)
• TIC-TAC: A Framework for Improved Covariance Estimation in Deep Heteroscedastic Regression [109.69084997173196]
  Deepscedastic regression involves jointly optimizing the mean and covariance of the predicted distribution using the negative log-likelihood. Recent works show that this may result in sub-optimal convergence due to the challenges associated with covariance estimation. We study two questions: (1) Does the predicted covariance truly capture the randomness of the predicted mean? Our results show that not only does TIC accurately learn the covariance, it additionally facilitates an improved convergence of the negative log-likelihood.
  arXiv  Detail & Related papers  (2023-10-29T09:54:03Z)
• Adaptive learning of density ratios in RKHS [3.047411947074805]
  Estimating the ratio of two probability densities from finitely many observations is a central problem in machine learning and statistics. We analyze a large class of density ratio estimation methods that minimize a regularized Bregman divergence between the true density ratio and a model in a reproducing kernel Hilbert space.
  arXiv  Detail & Related papers  (2023-07-30T08:18:39Z)
• Adapting to Continuous Covariate Shift via Online Density Ratio Estimation [64.8027122329609]
  Dealing with distribution shifts is one of the central challenges for modern machine learning. We propose an online method that can appropriately reuse historical information. Our density ratio estimation method is proven to perform well by enjoying a dynamic regret bound.
  arXiv  Detail & Related papers  (2023-02-06T04:03:33Z)
• Increasing the efficiency of randomized trial estimates via linear adjustment for a prognostic score [59.75318183140857]
  Estimating causal effects from randomized experiments is central to clinical research. Most methods for historical borrowing achieve reductions in variance by sacrificing strict type-I error rate control.
  arXiv  Detail & Related papers  (2020-12-17T21:10:10Z)
• Improving Nonparametric Density Estimation with Tensor Decompositions [14.917420021212912]
  Nonparametric density estimators often perform well on low dimensional data, but suffer when applied to higher dimensional data. This paper investigates whether these improvements can be extended to other simplified dependence assumptions. We prove that restricting estimation to low-rank nonnegative PARAFAC or Tucker decompositions removes the dimensionality exponent on bin width rates for multidimensional histograms.
  arXiv  Detail & Related papers  (2020-10-06T01:39:09Z)
• TraDE: Transformers for Density Estimation [101.20137732920718]
  TraDE is a self-attention-based architecture for auto-regressive density estimation. We present a suite of tasks such as regression using generated samples, out-of-distribution detection, and robustness to noise in the training data.
  arXiv  Detail & Related papers  (2020-04-06T07:32:51Z)
• Posterior Ratio Estimation of Latent Variables [14.619879849533662]
  In some applications, we want to compare distributions of random variables that are emphinferred from observations. We study the problem of estimating the ratio between two posterior probability density functions of a latent variable.
  arXiv  Detail & Related papers  (2020-02-15T16:46:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.