Generalized Resubstitution for Classification Error Estimation

• URL: http://arxiv.org/abs/2110.12285v1
• Date: Sat, 23 Oct 2021 19:42:11 GMT
• Title: Generalized Resubstitution for Classification Error Estimation
• Authors: Parisa Ghane and Ulisses Braga-Neto
• Abstract summary: We propose the family of generalized resubstitution error estimators based on empirical measures. These error estimators are computationally efficient and do not require re-training of classifiers.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We propose the family of generalized resubstitution classifier error estimators based on empirical measures. These error estimators are computationally efficient and do not require re-training of classifiers. The plain resubstitution error estimator corresponds to choosing the standard empirical measure. Other choices of empirical measure lead to bolstered, posterior-probability, Gaussian-process, and Bayesian error estimators; in addition, we propose bolstered posterior-probability error estimators as a new family of generalized resubstitution estimators. In the two-class case, we show that a generalized resubstitution estimator is consistent and asymptotically unbiased, regardless of the distribution of the features and label, if the corresponding generalized empirical measure converges uniformly to the standard empirical measure and the classification rule has a finite VC dimension. A generalized resubstitution estimator typically has hyperparameters that can be tuned to control its bias and variance, which adds flexibility. Numerical experiments with various classification rules trained on synthetic data assess the thefinite-sample performance of several representative generalized resubstitution error estimators. In addition, results of an image classification experiment using the LeNet-5 convolutional neural network and the MNIST data set demonstrate the potential of this class of error estimators in deep learning for computer vision.

Related papers

• Generalized Resubstitution for Regression Error Estimation [1.6114012813668932]
  We propose resubstitution error estimators for regression corresponding to a choice of empirical probability measures and loss function. We prove that these error estimators are consistent under broad assumptions.
  arXiv  Detail & Related papers  (2024-10-23T15:22:21Z)
• Estimating Generalization Performance Along the Trajectory of Proximal SGD in Robust Regression [4.150180443030652]
  We introduce estimators that precisely track the generalization error of the iterates along the trajectory of the iterative algorithm. The results are illustrated through several examples, including Huber regression, pseudo-Huber regression, and their penalized variants with non-smooth regularizer.
  arXiv  Detail & Related papers  (2024-10-03T16:13:42Z)
• Generalization bounds for regression and classification on adaptive covering input domains [1.4141453107129398]
  We focus on the generalization bound, which serves as an upper limit for the generalization error. In the case of classification tasks, we treat the target function as a one-hot, a piece-wise constant function, and employ 0/1 loss for error measurement.
  arXiv  Detail & Related papers  (2024-07-29T05:40:08Z)
• Deep Imbalanced Regression via Hierarchical Classification Adjustment [50.19438850112964]
  Regression tasks in computer vision are often formulated into classification by quantizing the target space into classes. The majority of training samples lie in a head range of target values, while a minority of samples span a usually larger tail range. We propose to construct hierarchical classifiers for solving imbalanced regression tasks. Our novel hierarchical classification adjustment (HCA) for imbalanced regression shows superior results on three diverse tasks.
  arXiv  Detail & Related papers  (2023-10-26T04:54:39Z)
• Corrected generalized cross-validation for finite ensembles of penalized estimators [5.165142221427927]
  Generalized cross-validation (GCV) is a widely-used method for estimating the squared out-of-sample prediction risk. We show that GCV is inconsistent for any finite ensemble of size greater than one.
  arXiv  Detail & Related papers  (2023-10-02T17:38:54Z)
• A Statistical Model for Predicting Generalization in Few-Shot Classification [6.158812834002346]
  We introduce a Gaussian model of the feature distribution to predict the generalization error. We show that our approach outperforms alternatives such as the leave-one-out cross-validation strategy.
  arXiv  Detail & Related papers  (2022-12-13T10:21:15Z)
• Predicting Unreliable Predictions by Shattering a Neural Network [145.3823991041987]
  Piecewise linear neural networks can be split into subfunctions. Subfunctions have their own activation pattern, domain, and empirical error. Empirical error for the full network can be written as an expectation over subfunctions.
  arXiv  Detail & Related papers  (2021-06-15T18:34:41Z)
• Evaluating State-of-the-Art Classification Models Against Bayes Optimality [106.50867011164584]
  We show that we can compute the exact Bayes error of generative models learned using normalizing flows. We use our approach to conduct a thorough investigation of state-of-the-art classification models.
  arXiv  Detail & Related papers  (2021-06-07T06:21:20Z)
• Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
  We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers. We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model. Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
  arXiv  Detail & Related papers  (2020-06-22T21:12:31Z)
• Performance metrics for intervention-triggering prediction models do not reflect an expected reduction in outcomes from using the model [71.9860741092209]
  Clinical researchers often select among and evaluate risk prediction models. Standard metrics calculated from retrospective data are only related to model utility under certain assumptions. When predictions are delivered repeatedly throughout time, the relationship between standard metrics and utility is further complicated.
  arXiv  Detail & Related papers  (2020-06-02T16:26:49Z)
• Estimating Gradients for Discrete Random Variables by Sampling without Replacement [93.09326095997336]
  We derive an unbiased estimator for expectations over discrete random variables based on sampling without replacement. We show that our estimator can be derived as the Rao-Blackwellization of three different estimators.
  arXiv  Detail & Related papers  (2020-02-14T14:15:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.