Mitigating multiple descents: A model-agnostic framework for risk
monotonization
- URL: http://arxiv.org/abs/2205.12937v1
- Date: Wed, 25 May 2022 17:41:40 GMT
- Title: Mitigating multiple descents: A model-agnostic framework for risk
monotonization
- Authors: Pratik Patil, Arun Kumar Kuchibhotla, Yuting Wei, Alessandro Rinaldo
- Abstract summary: We develop a general framework for risk monotonization based on cross-validation.
We propose two data-driven methodologies, namely zero- and one-step, that are akin to bagging and boosting.
- Score: 84.6382406922369
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recent empirical and theoretical analyses of several commonly used prediction
procedures reveal a peculiar risk behavior in high dimensions, referred to as
double/multiple descent, in which the asymptotic risk is a non-monotonic
function of the limiting aspect ratio of the number of features or parameters
to the sample size. To mitigate this undesirable behavior, we develop a general
framework for risk monotonization based on cross-validation that takes as input
a generic prediction procedure and returns a modified procedure whose
out-of-sample prediction risk is, asymptotically, monotonic in the limiting
aspect ratio. As part of our framework, we propose two data-driven
methodologies, namely zero- and one-step, that are akin to bagging and
boosting, respectively, and show that, under very mild assumptions, they
provably achieve monotonic asymptotic risk behavior. Our results are applicable
to a broad variety of prediction procedures and loss functions, and do not
require a well-specified (parametric) model. We exemplify our framework with
concrete analyses of the minimum $\ell_2$, $\ell_1$-norm least squares
prediction procedures. As one of the ingredients in our analysis, we also
derive novel additive and multiplicative forms of oracle risk inequalities for
split cross-validation that are of independent interest.
Related papers
- Precise analysis of ridge interpolators under heavy correlations -- a Random Duality Theory view [0.0]
We show that emphRandom Duality Theory (RDT) can be utilized to obtain precise closed form characterizations of all estimators related optimizing quantities of interest.
arXiv Detail & Related papers (2024-06-13T14:56:52Z) - Selective Nonparametric Regression via Testing [54.20569354303575]
We develop an abstention procedure via testing the hypothesis on the value of the conditional variance at a given point.
Unlike existing methods, the proposed one allows to account not only for the value of the variance itself but also for the uncertainty of the corresponding variance predictor.
arXiv Detail & Related papers (2023-09-28T13:04:11Z) - Prediction Risk and Estimation Risk of the Ridgeless Least Squares Estimator under General Assumptions on Regression Errors [10.857775300638831]
We explore prediction risk as well as estimation risk under more general regression error assumptions.
Our findings suggest that the benefits of over parameterization can extend to time series, panel and grouped data.
arXiv Detail & Related papers (2023-05-22T10:04:20Z) - Vector-Valued Least-Squares Regression under Output Regularity
Assumptions [73.99064151691597]
We propose and analyse a reduced-rank method for solving least-squares regression problems with infinite dimensional output.
We derive learning bounds for our method, and study under which setting statistical performance is improved in comparison to full-rank method.
arXiv Detail & Related papers (2022-11-16T15:07:00Z) - Bagging in overparameterized learning: Risk characterization and risk
monotonization [2.6534407766508177]
We study the prediction risk of variants of bagged predictors under the proportionals regime.
Specifically, we propose a general strategy to analyze the prediction risk under squared error loss of bagged predictors.
arXiv Detail & Related papers (2022-10-20T17:45:58Z) - Predictive Multiplicity in Probabilistic Classification [25.111463701666864]
We present a framework for measuring predictive multiplicity in probabilistic classification.
We demonstrate the incidence and prevalence of predictive multiplicity in real-world tasks.
Our results emphasize the need to report predictive multiplicity more widely.
arXiv Detail & Related papers (2022-06-02T16:25:29Z) - Empirical or Invariant Risk Minimization? A Sample Complexity
Perspective [49.43806345820883]
It is unclear when in-variant risk generalization (IRM) should be preferred over the widely-employed empirical risk minimization (ERM) framework.
We find that depending on the type of data generation mechanism, the two approaches might have very different finite sample and behavior.
We further investigate how different factors -- the number of environments, complexity of the model, and IRM penalty weight -- impact the sample complexity of IRM in relation to its distance from the OOD solutions.
arXiv Detail & Related papers (2020-10-30T17:55:30Z) - The Risks of Invariant Risk Minimization [52.7137956951533]
Invariant Risk Minimization is an objective based on the idea for learning deep, invariant features of data.
We present the first analysis of classification under the IRM objective--as well as these recently proposed alternatives--under a fairly natural and general model.
We show that IRM can fail catastrophically unless the test data are sufficiently similar to the training distribution--this is precisely the issue that it was intended to solve.
arXiv Detail & Related papers (2020-10-12T14:54:32Z) - A One-step Approach to Covariate Shift Adaptation [82.01909503235385]
A default assumption in many machine learning scenarios is that the training and test samples are drawn from the same probability distribution.
We propose a novel one-step approach that jointly learns the predictive model and the associated weights in one optimization.
arXiv Detail & Related papers (2020-07-08T11:35:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.