Conformal Predictions for Probabilistically Robust Scalable Machine Learning Classification

• URL: http://arxiv.org/abs/2403.10368v1
• Date: Fri, 15 Mar 2024 14:59:24 GMT
• Title: Conformal Predictions for Probabilistically Robust Scalable Machine Learning Classification
• Authors: Alberto Carlevaro, Teodoro Alamo Cantarero, Fabrizio Dabbene, Maurizio Mongelli,
• Abstract summary: Conformal predictions make it possible to define reliable and robust learning algorithms. They are essentially a method for evaluating whether an algorithm is good enough to be used in practice. This paper defines a reliable learning framework for classification from the very beginning of its design.
• Score: 1.757077789361314
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Conformal predictions make it possible to define reliable and robust learning algorithms. But they are essentially a method for evaluating whether an algorithm is good enough to be used in practice. To define a reliable learning framework for classification from the very beginning of its design, the concept of scalable classifier was introduced to generalize the concept of classical classifier by linking it to statistical order theory and probabilistic learning theory. In this paper, we analyze the similarities between scalable classifiers and conformal predictions by introducing a new definition of a score function and defining a special set of input variables, the conformal safety set, which can identify patterns in the input space that satisfy the error coverage guarantee, i.e., that the probability of observing the wrong (possibly unsafe) label for points belonging to this set is bounded by a predefined $\varepsilon$ error level. We demonstrate the practical implications of this framework through an application in cybersecurity for identifying DNS tunneling attacks. Our work contributes to the development of probabilistically robust and reliable machine learning models.

Related papers

• Trustworthy Classification through Rank-Based Conformal Prediction Sets [9.559062601251464]
  We propose a novel conformal prediction method that employs a rank-based score function suitable for classification models. Our approach constructs prediction sets that achieve the desired coverage rate while managing their size. Our contributions include a novel conformal prediction method, theoretical analysis, and empirical evaluation.
  arXiv  Detail & Related papers  (2024-07-05T10:43:41Z)
• Probabilistic Safety Regions Via Finite Families of Scalable Classifiers [2.431537995108158]
  Supervised classification recognizes patterns in the data to separate classes of behaviours. Canonical solutions contain misclassification errors that are intrinsic to the numerical approximating nature of machine learning. We introduce the concept of probabilistic safety region to describe a subset of the input space in which the number of misclassified instances is probabilistically controlled.
  arXiv  Detail & Related papers  (2023-09-08T22:40:19Z)
• When Does Confidence-Based Cascade Deferral Suffice? [69.28314307469381]
  Cascades are a classical strategy to enable inference cost to vary adaptively across samples. A deferral rule determines whether to invoke the next classifier in the sequence, or to terminate prediction. Despite being oblivious to the structure of the cascade, confidence-based deferral often works remarkably well in practice.
  arXiv  Detail & Related papers  (2023-07-06T04:13:57Z)
• Provable concept learning for interpretable predictions using variational inference [7.0349768355860895]
  In safety critical applications, practitioners are reluctant to trust neural networks when no interpretable explanations are available. We propose a probabilistic modeling framework to derive (C)oncept (L)earning and (P)rediction (CLAP) We prove that our method is able to identify them while attaining optimal classification accuracy.
  arXiv  Detail & Related papers  (2022-04-01T14:51:38Z)
• NUQ: Nonparametric Uncertainty Quantification for Deterministic Neural Networks [151.03112356092575]
  We show the principled way to measure the uncertainty of predictions for a classifier based on Nadaraya-Watson's nonparametric estimate of the conditional label distribution. We demonstrate the strong performance of the method in uncertainty estimation tasks on a variety of real-world image datasets.
  arXiv  Detail & Related papers  (2022-02-07T12:30:45Z)
• CC-Cert: A Probabilistic Approach to Certify General Robustness of Neural Networks [58.29502185344086]
  In safety-critical machine learning applications, it is crucial to defend models against adversarial attacks. It is important to provide provable guarantees for deep learning models against semantically meaningful input transformations. We propose a new universal probabilistic certification approach based on Chernoff-Cramer bounds.
  arXiv  Detail & Related papers  (2021-09-22T12:46:04Z)
• Trust but Verify: Assigning Prediction Credibility by Counterfactual Constrained Learning [123.3472310767721]
  Prediction credibility measures are fundamental in statistics and machine learning. These measures should account for the wide variety of models used in practice. The framework developed in this work expresses the credibility as a risk-fit trade-off.
  arXiv  Detail & Related papers  (2020-11-24T19:52:38Z)
• Failure Prediction by Confidence Estimation of Uncertainty-Aware Dirichlet Networks [6.700873164609009]
  It is shown that uncertainty-aware deep Dirichlet neural networks provide an improved separation between the confidence of correct and incorrect predictions in the true class probability metric. A new criterion is proposed for learning the true class probability by matching prediction confidence scores while taking imbalance and TCP constraints into account.
  arXiv  Detail & Related papers  (2020-10-19T21:06:45Z)
• Evaluating probabilistic classifiers: Reliability diagrams and score decompositions revisited [68.8204255655161]
  We introduce the CORP approach, which generates provably statistically Consistent, Optimally binned, and Reproducible reliability diagrams in an automated way. Corpor is based on non-parametric isotonic regression and implemented via the Pool-adjacent-violators (PAV) algorithm.
  arXiv  Detail & Related papers  (2020-08-07T08:22:26Z)
• Certified Robustness to Label-Flipping Attacks via Randomized Smoothing [105.91827623768724]
  Machine learning algorithms are susceptible to data poisoning attacks. We present a unifying view of randomized smoothing over arbitrary functions. We propose a new strategy for building classifiers that are pointwise-certifiably robust to general data poisoning attacks.
  arXiv  Detail & Related papers  (2020-02-07T21:28:30Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.