Covariate Shift in High-Dimensional Random Feature Regression
- URL: http://arxiv.org/abs/2111.08234v1
- Date: Tue, 16 Nov 2021 05:23:28 GMT
- Title: Covariate Shift in High-Dimensional Random Feature Regression
- Authors: Nilesh Tripuraneni, Ben Adlam, Jeffrey Pennington
- Abstract summary: Covariate shift is a significant obstacle in the development of robust machine learning models.
We present a theoretical understanding in context of modern machine learning.
- Score: 44.13449065077103
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: A significant obstacle in the development of robust machine learning models
is covariate shift, a form of distribution shift that occurs when the input
distributions of the training and test sets differ while the conditional label
distributions remain the same. Despite the prevalence of covariate shift in
real-world applications, a theoretical understanding in the context of modern
machine learning has remained lacking. In this work, we examine the exact
high-dimensional asymptotics of random feature regression under covariate shift
and present a precise characterization of the limiting test error, bias, and
variance in this setting. Our results motivate a natural partial order over
covariate shifts that provides a sufficient condition for determining when the
shift will harm (or even help) test performance. We find that overparameterized
models exhibit enhanced robustness to covariate shift, providing one of the
first theoretical explanations for this intriguing phenomenon. Additionally,
our analysis reveals an exact linear relationship between in-distribution and
out-of-distribution generalization performance, offering an explanation for
this surprising recent empirical observation.
Related papers
- Generalization vs. Specialization under Concept Shift [12.196508752999797]
Machine learning models are often brittle under distribution shift.
We show that test performance can exhibit a nonmonotonic data dependence, even when double descent is absent.
Experiments on MNIST and FashionMNIST suggest that this intriguing behavior is present also in classification problems.
arXiv Detail & Related papers (2024-09-23T22:30:28Z) - Identifiable Latent Neural Causal Models [82.14087963690561]
Causal representation learning seeks to uncover latent, high-level causal representations from low-level observed data.
We determine the types of distribution shifts that do contribute to the identifiability of causal representations.
We translate our findings into a practical algorithm, allowing for the acquisition of reliable latent causal representations.
arXiv Detail & Related papers (2024-03-23T04:13:55Z) - Invariant Anomaly Detection under Distribution Shifts: A Causal
Perspective [6.845698872290768]
Anomaly detection (AD) is the machine learning task of identifying highly discrepant abnormal samples.
Under the constraints of a distribution shift, the assumption that training samples and test samples are drawn from the same distribution breaks down.
We attempt to increase the resilience of anomaly detection models to different kinds of distribution shifts.
arXiv Detail & Related papers (2023-12-21T23:20:47Z) - Blessings and Curses of Covariate Shifts: Adversarial Learning Dynamics, Directional Convergence, and Equilibria [6.738946307589742]
Covariate distribution shifts and adversarial perturbations present challenges to the conventional statistical learning framework.
This paper precisely characterizes the extrapolation region, examining both regression and classification in an infinite-dimensional setting.
arXiv Detail & Related papers (2022-12-05T18:00:31Z) - Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics
for Convex Losses in High-Dimension [25.711297863946193]
We develop a theory for the study of fluctuations in an ensemble of generalised linear models trained on different, but correlated, features.
We provide a complete description of the joint distribution of the empirical risk minimiser for generic convex loss and regularisation in the high-dimensional limit.
arXiv Detail & Related papers (2022-01-31T17:44:58Z) - Certifying Model Accuracy under Distribution Shifts [151.67113334248464]
We present provable robustness guarantees on the accuracy of a model under bounded Wasserstein shifts of the data distribution.
We show that a simple procedure that randomizes the input of the model within a transformation space is provably robust to distributional shifts under the transformation.
arXiv Detail & Related papers (2022-01-28T22:03:50Z) - The Hidden Uncertainty in a Neural Networks Activations [105.4223982696279]
The distribution of a neural network's latent representations has been successfully used to detect out-of-distribution (OOD) data.
This work investigates whether this distribution correlates with a model's epistemic uncertainty, thus indicating its ability to generalise to novel inputs.
arXiv Detail & Related papers (2020-12-05T17:30:35Z) - A One-step Approach to Covariate Shift Adaptation [82.01909503235385]
A default assumption in many machine learning scenarios is that the training and test samples are drawn from the same probability distribution.
We propose a novel one-step approach that jointly learns the predictive model and the associated weights in one optimization.
arXiv Detail & Related papers (2020-07-08T11:35:47Z) - Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers.
We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model.
Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
arXiv Detail & Related papers (2020-06-22T21:12:31Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.