Towards a Unified Analysis of Kernel-based Methods Under Covariate Shift
- URL: http://arxiv.org/abs/2310.08237v2
- Date: Thu, 19 Oct 2023 07:24:47 GMT
- Title: Towards a Unified Analysis of Kernel-based Methods Under Covariate Shift
- Authors: Xingdong Feng, Xin He, Caixing Wang, Chao Wang, Jingnan Zhang
- Abstract summary: We propose a unified analysis of general nonparametric methods in a reproducing kernel Hilbert space.
Our theoretical results are established for a general loss belonging to a rich loss function family.
Our results concur with the optimal results in literature where the squared loss is used.
- Score: 18.240776405802205
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Covariate shift occurs prevalently in practice, where the input distributions
of the source and target data are substantially different. Despite its
practical importance in various learning problems, most of the existing methods
only focus on some specific learning tasks and are not well validated
theoretically and numerically. To tackle this problem, we propose a unified
analysis of general nonparametric methods in a reproducing kernel Hilbert space
(RKHS) under covariate shift. Our theoretical results are established for a
general loss belonging to a rich loss function family, which includes many
commonly used methods as special cases, such as mean regression, quantile
regression, likelihood-based classification, and margin-based classification.
Two types of covariate shift problems are the focus of this paper and the sharp
convergence rates are established for a general loss function to provide a
unified theoretical analysis, which concurs with the optimal results in
literature where the squared loss is used. Extensive numerical studies on
synthetic and real examples confirm our theoretical findings and further
illustrate the effectiveness of our proposed method.
Related papers
- On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics.
The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
arXiv Detail & Related papers (2023-12-13T02:11:07Z) - A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment
for Imbalanced Learning [129.63326990812234]
We propose a technique named data-dependent contraction to capture how modified losses handle different classes.
On top of this technique, a fine-grained generalization bound is established for imbalanced learning, which helps reveal the mystery of re-weighting and logit-adjustment.
arXiv Detail & Related papers (2023-10-07T09:15:08Z) - Inconsistency, Instability, and Generalization Gap of Deep Neural
Network Training [14.871738070617491]
We show that inconsistency is a more reliable indicator of generalization gap than the sharpness of the loss landscape.
The results also provide a theoretical basis for existing methods such as co-distillation and ensemble.
arXiv Detail & Related papers (2023-05-31T20:28:13Z) - Hypothesis Transfer Learning with Surrogate Classification Losses:
Generalization Bounds through Algorithmic Stability [3.908842679355255]
Hypothesis transfer learning (HTL) contrasts domain adaptation by allowing for a previous task leverage, named the source, into a new one, the target.
This paper studies the learning theory of HTL through algorithmic stability, an attractive theoretical framework for machine learning algorithms analysis.
arXiv Detail & Related papers (2023-05-31T09:38:21Z) - SARAH-based Variance-reduced Algorithm for Stochastic Finite-sum
Cocoercive Variational Inequalities [137.6408511310322]
We consider the problem of finite-sum cocoercive variational inequalities.
For strongly monotone problems it is possible to achieve linear convergence to a solution using this method.
arXiv Detail & Related papers (2022-10-12T08:04:48Z) - Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics
for Convex Losses in High-Dimension [25.711297863946193]
We develop a theory for the study of fluctuations in an ensemble of generalised linear models trained on different, but correlated, features.
We provide a complete description of the joint distribution of the empirical risk minimiser for generic convex loss and regularisation in the high-dimensional limit.
arXiv Detail & Related papers (2022-01-31T17:44:58Z) - A Theoretical Analysis on Independence-driven Importance Weighting for
Covariate-shift Generalization [44.88645911638269]
independence-driven importance algorithms in stable learning literature have shown empirical effectiveness.
In this paper, we theoretically prove the effectiveness of such algorithms by explaining them as feature selection processes.
We prove that under ideal conditions, independence-driven importance weighting algorithms could identify the variables in this set.
arXiv Detail & Related papers (2021-11-03T17:18:49Z) - A One-step Approach to Covariate Shift Adaptation [82.01909503235385]
A default assumption in many machine learning scenarios is that the training and test samples are drawn from the same probability distribution.
We propose a novel one-step approach that jointly learns the predictive model and the associated weights in one optimization.
arXiv Detail & Related papers (2020-07-08T11:35:47Z) - Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers.
We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model.
Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
arXiv Detail & Related papers (2020-06-22T21:12:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.