Robustness quantification: a new method for assessing the reliability of the predictions of a classifier

• URL: http://arxiv.org/abs/2503.22418v2
• Date: Thu, 10 Apr 2025 02:05:10 GMT
• Title: Robustness quantification: a new method for assessing the reliability of the predictions of a classifier
• Authors: Adrián Detavernier, Jasper De Bock,
• Abstract summary: Based on existing ideas in the field of imprecise probabilities, we present a new approach for assessing the reliability of the individual predictions of a generative probabilistic classifier.<n>We call this approach robustness quantification, compare it to uncertainty quantification, and demonstrate that it continues to work well even for classifiers that are learned from small training sets that are sampled from a shifted distribution.
• Score: 0.14732811715354452
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Based on existing ideas in the field of imprecise probabilities, we present a new approach for assessing the reliability of the individual predictions of a generative probabilistic classifier. We call this approach robustness quantification, compare it to uncertainty quantification, and demonstrate that it continues to work well even for classifiers that are learned from small training sets that are sampled from a shifted distribution.

Related papers

• Trustworthy Classification through Rank-Based Conformal Prediction Sets [9.559062601251464]
  We propose a novel conformal prediction method that employs a rank-based score function suitable for classification models. Our approach constructs prediction sets that achieve the desired coverage rate while managing their size. Our contributions include a novel conformal prediction method, theoretical analysis, and empirical evaluation.
  arXiv  Detail & Related papers  (2024-07-05T10:43:41Z)
• Uncertainty Quantification via Stable Distribution Propagation [60.065272548502]
  We propose a new approach for propagating stable probability distributions through neural networks. Our method is based on local linearization, which we show to be an optimal approximation in terms of total variation distance for the ReLU non-linearity.
  arXiv  Detail & Related papers  (2024-02-13T09:40:19Z)
• Quantification of Predictive Uncertainty via Inference-Time Sampling [57.749601811982096]
  We propose a post-hoc sampling strategy for estimating predictive uncertainty accounting for data ambiguity. The method can generate different plausible outputs for a given input and does not assume parametric forms of predictive distributions.
  arXiv  Detail & Related papers  (2023-08-03T12:43:21Z)
• Variational Classification [51.2541371924591]
  We derive a variational objective to train the model, analogous to the evidence lower bound (ELBO) used to train variational auto-encoders. Treating inputs to the softmax layer as samples of a latent variable, our abstracted perspective reveals a potential inconsistency. We induce a chosen latent distribution, instead of the implicit assumption found in a standard softmax layer.
  arXiv  Detail & Related papers  (2023-05-17T17:47:19Z)
• Bayesian Quantification with Black-Box Estimators [1.599072005190786]
  Approaches like adjusted classify and count, black-box shift estimators, and invariant ratio estimators use an auxiliary (and potentially biased) black-box classifier to estimate the class distribution and yield guarantees under weak assumptions. We demonstrate that all these algorithms are closely related to the inference in a particular Bayesian Chain model, approxing the assumed ground-truthgenerative process. Then, we discuss an efficient Markov Monte Carlo sampling scheme for the introduced model and show an consistency guarantee in the large-data limit.
  arXiv  Detail & Related papers  (2023-02-17T22:10:04Z)
• Confidence-aware Training of Smoothed Classifiers for Certified Robustness [75.95332266383417]
  We use "accuracy under Gaussian noise" as an easy-to-compute proxy of adversarial robustness for an input. Our experiments show that the proposed method consistently exhibits improved certified robustness upon state-of-the-art training methods.
  arXiv  Detail & Related papers  (2022-12-18T03:57:12Z)
• ProBoost: a Boosting Method for Probabilistic Classifiers [55.970609838687864]
  ProBoost is a new boosting algorithm for probabilistic classifiers. It uses the uncertainty of each training sample to determine the most challenging/uncertain ones. It produces a sequence that progressively focuses on the samples found to have the highest uncertainty.
  arXiv  Detail & Related papers  (2022-09-04T12:49:20Z)
• Predictive Inference with Weak Supervision [3.1925030748447747]
  We bridge the gap between partial supervision and validation by developing a conformal prediction framework. We introduce a new notion of coverage and predictive validity, then develop several application scenarios. We corroborate the hypothesis that the new coverage definition allows for tighter and more informative (but valid) confidence sets.
  arXiv  Detail & Related papers  (2022-01-20T17:26:52Z)
• Credal Self-Supervised Learning [0.0]
  We show how to let the learner generate "pseudo-supervision" for unlabeled instances. In combination with consistency regularization, pseudo-labeling has shown promising performance in various domains. We compare our methodology to state-of-the-art self-supervision approaches.
  arXiv  Detail & Related papers  (2021-06-22T15:19:04Z)
• Entropy methods for the confidence assessment of probabilistic classification models [0.0]
  We argue that part of the information that is discarded in this process can be used to further evaluate the goodness of models. We provide a theoretical explanation of a confidence degradation phenomenon observed in the complement approach to the (Bernoulli) Naive Bayes generative model.
  arXiv  Detail & Related papers  (2021-03-28T15:39:13Z)
• Trusted Multi-View Classification [76.73585034192894]
  We propose a novel multi-view classification method, termed trusted multi-view classification. It provides a new paradigm for multi-view learning by dynamically integrating different views at an evidence level. The proposed algorithm jointly utilizes multiple views to promote both classification reliability and robustness.
  arXiv  Detail & Related papers  (2021-02-03T13:30:26Z)
• Out-of-distribution detection for regression tasks: parameter versus predictor entropy [2.026281591452464]
  It is crucial to detect when an instance lies downright too far from the training samples for the machine learning model to be trusted. For neural networks, one approach to this task consists of learning a diversity of predictors that all can explain the training data. We propose a new way of estimating the entropy of a distribution on predictors based on nearest neighbors in function space.
  arXiv  Detail & Related papers  (2020-10-24T21:41:21Z)
• Probabilistic Deep Learning for Instance Segmentation [9.62543698736491]
  We propose a generic method to obtain model-inherent uncertainty estimates within proposal-free instance segmentation models. We evaluate our method on the BBBC010 C. elegans dataset, where it yields competitive performance.
  arXiv  Detail & Related papers  (2020-08-24T19:51:48Z)
• Evaluating probabilistic classifiers: Reliability diagrams and score decompositions revisited [68.8204255655161]
  We introduce the CORP approach, which generates provably statistically Consistent, Optimally binned, and Reproducible reliability diagrams in an automated way. Corpor is based on non-parametric isotonic regression and implemented via the Pool-adjacent-violators (PAV) algorithm.
  arXiv  Detail & Related papers  (2020-08-07T08:22:26Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• Calibrated Prediction with Covariate Shift via Unsupervised Domain Adaptation [25.97333838935589]
  Uncertainty estimates are an important tool for helping autonomous agents or human decision makers understand and leverage predictive models. Existing algorithms can overestimate certainty, possibly yielding a false sense of confidence in the predictive model.
  arXiv  Detail & Related papers  (2020-02-29T20:31:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.