Learning under Target Shift: Optimal Density Ratio Estimation and Importance-Weighted Regression

• URL: http://arxiv.org/abs/2609.15785v1
• Date: Mon, 14 Sep 2026 16:05:44 GMT
• Title: Learning under Target Shift: Optimal Density Ratio Estimation and Importance-Weighted Regression
• Abstract summary: We study density ratio estimation and importance-weighted regression under target shift with continuous outputs.<n>We propose a spectral regularization method in a reproducing kernel Hilbert space (RKHS) for estimating the continuous density ratio.<n>We then incorporate the estimated density ratio into importance-weighted regression and characterize the propagation of density-ratio estimation error to the final predictor.
• Score: 0.2578242050187029
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We study density ratio estimation and importance-weighted regression under target shift with continuous outputs. Under target shift, the conditional distribution of the inputs given the outputs remains invariant across the training and test distributions, while the output marginal distribution may change. Although this problem has been extensively studied for discrete outputs, the continuous setting is substantially less understood: the importance weights are determined by an unknown density ratio function, for which existing estimation methods lack explicit finite-sample convergence rates. We propose a spectral regularization method in a reproducing kernel Hilbert space (RKHS) for estimating the continuous density ratio from labeled training samples and unlabeled test inputs. Under a source condition with regularity parameter $ι>0$, we establish high-probability finite-sample guarantees and show that the estimator achieves the capacity-independent minimax-optimal RKHS-norm rate $O(n_η^{-ι/(2ι+2)})$. We then incorporate the estimated density ratio into importance-weighted regression and characterize the propagation of density-ratio estimation error to the final predictor. When sufficiently many samples are available for density ratio estimation, the resulting regression estimator attains the minimax-optimal rates of standard kernel regression. These results establish a finite-sample theory for continuous density ratio estimation and importance-weighted learning under target shift.

Related papers

• Nonparametric Goodness-of-fit Testing under Covariate Shift [4.3686879183724665]
  The distribution mismatch is quantified by either a bounded moment condition or a sub-exponential tail condition on the target-to-source density ratio.<n>Our method combines truncated importance-weighting kernel ridge regression with a multiplier bootstrap to construct confidence sets for the regression function.<n>We prove nonasymptotic validity and sharpness of the resulting confidence sets under suitable operator compatibility conditions.
  arXiv  Detail & Related papers  (2026-08-05T13:51:43Z)
• Distributionally Faithful Imputation via Positive Semi-Definite Kernel Density Estimation [55.47411235954028]
  We recast imputation under missing completely at random as density estimation from masked observations.<n>We obtain a convex empirical risk problem with closed form marginals, solvable by a Newton interior point method.<n>The resulting PSD Impute model yields both single and multiple imputations from the same fitted density.
  arXiv  Detail & Related papers  (2026-07-08T15:51:05Z)
• Unbounded Density Ratio Estimation and Its Application to Covariate Shift Adaptation [6.455982311339433]
  unbounded density ratio estimation is an understudied yet critical challenge in statistical learning.<n>Much of the existing literature assumes that the density ratio is either uniformly bounded or unbounded but known exactly.<n>We propose a three-step estimation method that leverages unlabeled data from both the source and target distributions.
  arXiv  Detail & Related papers  (2026-03-31T13:24:50Z)
• Flow-Based Density Ratio Estimation for Intractable Distributions with Applications in Genomics [80.05951561886123]
  We leverage condition-aware flow matching to derive a single dynamical formulation for tracking density ratios along generative trajectories.<n>We demonstrate competitive performance on simulated benchmarks for closed-form ratio estimation, and show that our method supports versatile tasks in single-cell genomics data analysis.
  arXiv  Detail & Related papers  (2026-02-27T17:27:55Z)
• Estimating Unbounded Density Ratios: Applications in Error Control under Covariate Shift [17.924340241624204]
  We study density ratio estimators using loss functions based on at least squares and logistic regression.<n>We establish upper bounds on estimation errors with standard minimax optimal rates, up to logarithmic factors.<n>Our results accommodate density ratio functions with unbounded domains and ranges.
  arXiv  Detail & Related papers  (2025-03-29T11:35:39Z)
• Double Debiased Covariate Shift Adaptation Robust to Density-Ratio Estimation [7.8856737627153874]
  We propose a doubly robust estimator for covariate shift adaptation via importance weighting. Our estimator reduces the bias arising from the density ratio estimation errors. Notably, our estimator remains consistent if either the density ratio estimator or the regression function is consistent.
  arXiv  Detail & Related papers  (2023-10-25T13:38:29Z)
• Adaptive learning of density ratios in RKHS [3.047411947074805]
  Estimating the ratio of two probability densities from finitely many observations is a central problem in machine learning and statistics. We analyze a large class of density ratio estimation methods that minimize a regularized Bregman divergence between the true density ratio and a model in a reproducing kernel Hilbert space.
  arXiv  Detail & Related papers  (2023-07-30T08:18:39Z)
• Anomaly Detection with Variance Stabilized Density Estimation [49.46356430493534]
  We present a variance-stabilized density estimation problem for maximizing the likelihood of the observed samples. To obtain a reliable anomaly detector, we introduce a spectral ensemble of autoregressive models for learning the variance-stabilized distribution. We have conducted an extensive benchmark with 52 datasets, demonstrating that our method leads to state-of-the-art results.
  arXiv  Detail & Related papers  (2023-06-01T11:52:58Z)
• Adapting to Continuous Covariate Shift via Online Density Ratio Estimation [64.8027122329609]
  Dealing with distribution shifts is one of the central challenges for modern machine learning. We propose an online method that can appropriately reuse historical information. Our density ratio estimation method is proven to perform well by enjoying a dynamic regret bound.
  arXiv  Detail & Related papers  (2023-02-06T04:03:33Z)
• Heavy-tailed Streaming Statistical Estimation [58.70341336199497]
  We consider the task of heavy-tailed statistical estimation given streaming $p$ samples. We design a clipped gradient descent and provide an improved analysis under a more nuanced condition on the noise of gradients.
  arXiv  Detail & Related papers  (2021-08-25T21:30:27Z)
• TraDE: Transformers for Density Estimation [101.20137732920718]
  TraDE is a self-attention-based architecture for auto-regressive density estimation. We present a suite of tasks such as regression using generated samples, out-of-distribution detection, and robustness to noise in the training data.
  arXiv  Detail & Related papers  (2020-04-06T07:32:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.