Risk-based Calibration for Probabilistic Classifiers

• URL: http://arxiv.org/abs/2409.03542v1
• Date: Thu, 5 Sep 2024 14:06:56 GMT
• Title: Risk-based Calibration for Probabilistic Classifiers
• Authors: Aritz Pérez, Carlos Echegoyen, Guzmán Santafé,
• Abstract summary: We introduce a general iterative procedure called risk-based calibration (RC) to minimize the empirical risk under the 0-1 loss. RC improves the empirical error of the original closed-form learning algorithms and, more notably, consistently outperforms the gradient descent approach.
• Score: 4.792851066169872
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We introduce a general iterative procedure called risk-based calibration (RC) designed to minimize the empirical risk under the 0-1 loss (empirical error) for probabilistic classifiers. These classifiers are based on modeling probability distributions, including those constructed from the joint distribution (generative) and those based on the class conditional distribution (conditional). RC can be particularized to any probabilistic classifier provided a specific learning algorithm that computes the classifier's parameters in closed form using data statistics. RC reinforces the statistics aligned with the true class while penalizing those associated with other classes, guided by the 0-1 loss. The proposed method has been empirically tested on 30 datasets using na\"ive Bayes, quadratic discriminant analysis, and logistic regression classifiers. RC improves the empirical error of the original closed-form learning algorithms and, more notably, consistently outperforms the gradient descent approach with the three classifiers.

Related papers

• A method for classification of data with uncertainty using hypothesis testing [0.0]
  It is necessary to quantify uncertainty and adopt decision-making approaches that take it into account. We propose a new decision-making approach using two types of hypothesis testing. This method is capable of detecting ambiguous data that belong to the overlapping regions of two class distributions.
  arXiv  Detail & Related papers  (2025-02-12T17:14:07Z)
• A Hard-to-Beat Baseline for Training-free CLIP-based Adaptation [121.0693322732454]
  Contrastive Language-Image Pretraining (CLIP) has gained popularity for its remarkable zero-shot capacity. Recent research has focused on developing efficient fine-tuning methods to enhance CLIP's performance in downstream tasks. We revisit a classical algorithm, Gaussian Discriminant Analysis (GDA), and apply it to the downstream classification of CLIP.
  arXiv  Detail & Related papers  (2024-02-06T15:45:27Z)
• Probabilistic Safety Regions Via Finite Families of Scalable Classifiers [2.431537995108158]
  Supervised classification recognizes patterns in the data to separate classes of behaviours. Canonical solutions contain misclassification errors that are intrinsic to the numerical approximating nature of machine learning. We introduce the concept of probabilistic safety region to describe a subset of the input space in which the number of misclassified instances is probabilistically controlled.
  arXiv  Detail & Related papers  (2023-09-08T22:40:19Z)
• Complementary Labels Learning with Augmented Classes [22.460256396941528]
  Complementary Labels Learning (CLL) arises in many real-world tasks such as private questions classification and online learning. We propose a novel problem setting called Complementary Labels Learning with Augmented Classes (CLLAC) By using unlabeled data, we propose an unbiased estimator of classification risk for CLLAC, which is guaranteed to be provably consistent.
  arXiv  Detail & Related papers  (2022-11-19T13:55:27Z)
• Distributionally Robust Multiclass Classification and Applications in Deep Image Classifiers [9.979945269265627]
  We develop a Distributionally Robust Optimization (DRO) formulation for Multiclass Logistic Regression (MLR) We demonstrate reductions in test error rate by up to 83.5% and loss by up to 91.3% compared with baseline methods, by adopting a novel random training method.
  arXiv  Detail & Related papers  (2022-10-15T05:09:28Z)
• Learnable Distribution Calibration for Few-Shot Class-Incremental Learning [122.2241120474278]
  Few-shot class-incremental learning (FSCIL) faces challenges of memorizing old class distributions and estimating new class distributions given few training samples. We propose a learnable distribution calibration (LDC) approach, with the aim to systematically solve these two challenges using a unified framework.
  arXiv  Detail & Related papers  (2022-10-01T09:40:26Z)
• Learning from Multiple Unlabeled Datasets with Partial Risk Regularization [80.54710259664698]
  In this paper, we aim to learn an accurate classifier without any class labels. We first derive an unbiased estimator of the classification risk that can be estimated from the given unlabeled sets. We then find that the classifier obtained as such tends to cause overfitting as its empirical risks go negative during training. Experiments demonstrate that our method effectively mitigates overfitting and outperforms state-of-the-art methods for learning from multiple unlabeled sets.
  arXiv  Detail & Related papers  (2022-07-04T16:22:44Z)
• Self-Certifying Classification by Linearized Deep Assignment [65.0100925582087]
  We propose a novel class of deep predictors for classifying metric data on graphs within PAC-Bayes risk certification paradigm. Building on the recent PAC-Bayes literature and data-dependent priors, this approach enables learning posterior distributions on the hypothesis space.
  arXiv  Detail & Related papers  (2022-01-26T19:59:14Z)
• Shift Happens: Adjusting Classifiers [2.8682942808330703]
  Minimizing expected loss measured by a proper scoring rule, such as Brier score or log-loss (cross-entropy), is a common objective while training a probabilistic classifier. We propose methods that transform all predictions to (re)equalize the average prediction and the class distribution. We demonstrate experimentally that, when in practice the class distribution is known only approximately, there is often still a reduction in loss depending on the amount of shift and the precision to which the class distribution is known.
  arXiv  Detail & Related papers  (2021-11-03T21:27:27Z)
• When in Doubt: Improving Classification Performance with Alternating Normalization [57.39356691967766]
  We introduce Classification with Alternating Normalization (CAN), a non-parametric post-processing step for classification. CAN improves classification accuracy for challenging examples by re-adjusting their predicted class probability distribution. We empirically demonstrate its effectiveness across a diverse set of classification tasks.
  arXiv  Detail & Related papers  (2021-09-28T02:55:42Z)
• Distributionally Robust Multiclass Classification and Applications in Deep Image Classifiers [3.179831861897336]
  We develop a Distributionally Robust Optimization (DRO) formulation for Multiclass Logistic Regression (MLR) We demonstrate reductions in test error rate by up to 83.5% and loss by up to 91.3% compared with baseline methods, by adopting a novel random training method.
  arXiv  Detail & Related papers  (2021-09-27T02:58:19Z)
• Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
  We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers. We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model. Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
  arXiv  Detail & Related papers  (2020-06-22T21:12:31Z)
• VaB-AL: Incorporating Class Imbalance and Difficulty with Variational Bayes for Active Learning [38.33920705605981]
  We propose a method that can naturally incorporate class imbalance into the Active Learning framework. We show that our method can be applied to tasks classification on multiple different datasets.
  arXiv  Detail & Related papers  (2020-03-25T07:34:06Z)
• Certified Robustness to Label-Flipping Attacks via Randomized Smoothing [105.91827623768724]
  Machine learning algorithms are susceptible to data poisoning attacks. We present a unifying view of randomized smoothing over arbitrary functions. We propose a new strategy for building classifiers that are pointwise-certifiably robust to general data poisoning attacks.
  arXiv  Detail & Related papers  (2020-02-07T21:28:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.