Inductive Conformal Prediction: A Straightforward Introduction with Examples in Python

• URL: http://arxiv.org/abs/2206.11810v2
• Date: Fri, 24 Jun 2022 01:09:36 GMT
• Title: Inductive Conformal Prediction: A Straightforward Introduction with Examples in Python
• Authors: Martim Sousa
• Abstract summary: Inductive Conformal Prediction (ICP) is a set of distribution-free and model agnostic algorithms devised to predict with a user-defined confidence with coverage guarantee. ICP takes special importance in high-risk settings where we want the true output to belong to the prediction set with high probability. This paper is a hands-on introduction, this means that we will provide examples as we introduce the theory.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inductive Conformal Prediction (ICP) is a set of distribution-free and model agnostic algorithms devised to predict with a user-defined confidence with coverage guarantee. Instead of having point predictions, i.e., a real number in the case of regression or a single class in multi class classification, models calibrated using ICP output an interval or a set of classes, respectively. ICP takes special importance in high-risk settings where we want the true output to belong to the prediction set with high probability. As an example, a classification model might output that given a magnetic resonance image a patient has no latent diseases to report. However, this model output was based on the most likely class, the second most likely class might tell that the patient has a 15% chance of brain tumor or other severe disease and therefore further exams should be conducted. Using ICP is therefore way more informative and we believe that should be the standard way of producing forecasts. This paper is a hands-on introduction, this means that we will provide examples as we introduce the theory.

Related papers

• Classification with Reject Option: Distribution-free Error Guarantees via Conformal Prediction [1.1380162891529535]
  We formalise the approach to machine learning with reject option in binary classification.<n>We provide theoretical guarantees on the resulting error rate.<n>Error-reject curves illustrate the trade-off between error rate and reject rate.
  arXiv  Detail & Related papers  (2025-06-26T23:04:25Z)
• One Sample is Enough to Make Conformal Prediction Robust [53.78604391939934]
  We show that conformal prediction attains some robustness even with a forward pass on a single randomly perturbed input.<n>Our approach returns robust sets with smaller average set size compared to SOTA methods which use many (e.g. around 100) passes per input.
  arXiv  Detail & Related papers  (2025-06-19T19:14:25Z)
• Optimal Conformal Prediction under Epistemic Uncertainty [61.46247583794497]
  Conformal prediction (CP) is a popular framework for representing uncertainty.<n>We introduce Bernoulli prediction sets (BPS) which produce the smallest prediction sets that ensure conditional coverage.<n>When given first-order predictions, BPS reduces to the well-known adaptive prediction sets (APS)
  arXiv  Detail & Related papers  (2025-05-25T08:32:44Z)
• Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filtering [55.15192437680943]
  Generative models lack rigorous statistical guarantees for their outputs. We propose a sequential conformal prediction method producing prediction sets that satisfy a rigorous statistical guarantee. This guarantee states that with high probability, the prediction sets contain at least one admissible (or valid) example.
  arXiv  Detail & Related papers  (2024-10-02T15:26:52Z)
• On Temperature Scaling and Conformal Prediction of Deep Classifiers [9.975341265604577]
  Two popular approaches for that aim are: 1): modifies the classifier's softmax values such that the maximal value better estimates the correctness probability; and 2) Conformal Prediction (CP): produces a prediction set of candidate labels that contains the true label with a user-specified probability. In practice, both types of indications are desirable, yet, so far the interplay between them has not been investigated.
  arXiv  Detail & Related papers  (2024-02-08T16:45:12Z)
• Conformal Prediction for Deep Classifier via Label Ranking [29.784336674173616]
  Conformal prediction is a statistical framework that generates prediction sets with a desired coverage guarantee. We propose a novel algorithm named $textitSorted Adaptive Prediction Sets$ (SAPS) SAPS discards all the probability values except for the maximum softmax probability.
  arXiv  Detail & Related papers  (2023-10-10T08:54:14Z)
• Performance Metrics for Probabilistic Ordinal Classifiers [1.6653762541912462]
  Ordinal classification models assign higher penalties to predictions further away from the true class. This paper advocates the use of the Ranked Probability Score (RPS) for image grading tasks.
  arXiv  Detail & Related papers  (2023-09-15T18:45:15Z)
• RR-CP: Reliable-Region-Based Conformal Prediction for Trustworthy Medical Image Classification [24.52922162675259]
  Conformal prediction (CP) generates a set of predictions for a given test sample. The size of the set indicates how certain the predictions are. We propose a new method called Reliable-Region-Based Conformal Prediction (RR-CP)
  arXiv  Detail & Related papers  (2023-09-09T11:14:04Z)
• When Does Confidence-Based Cascade Deferral Suffice? [69.28314307469381]
  Cascades are a classical strategy to enable inference cost to vary adaptively across samples. A deferral rule determines whether to invoke the next classifier in the sequence, or to terminate prediction. Despite being oblivious to the structure of the cascade, confidence-based deferral often works remarkably well in practice.
  arXiv  Detail & Related papers  (2023-07-06T04:13:57Z)
• Conformal Language Modeling [61.94417935386489]
  We propose a novel approach to conformal prediction for generative language models (LMs) Standard conformal prediction produces prediction sets with rigorous, statistical guarantees. We demonstrate the promise of our approach on multiple tasks in open-domain question answering, text summarization, and radiology report generation.
  arXiv  Detail & Related papers  (2023-06-16T21:55:08Z)
• Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimation [51.809741427975105]
  This work aims to assess how well a model performs under distribution shifts without using labels. We use the nuclear norm that has been shown to be effective in characterizing both properties. We show that the nuclear norm is more accurate and robust in accuracy than existing methods.
  arXiv  Detail & Related papers  (2023-02-02T13:30:48Z)
• Probabilistic Conformal Prediction Using Conditional Random Samples [73.26753677005331]
  PCP is a predictive inference algorithm that estimates a target variable by a discontinuous predictive set. It is efficient and compatible with either explicit or implicit conditional generative models.
  arXiv  Detail & Related papers  (2022-06-14T03:58:03Z)
• A Confidence Machine for Sparse High-Order Interaction Model [16.780058676633914]
  Conformal prediction (CP) is a promising approach for obtaining the confidence of prediction results with fewer theoretical assumptions. We develop a full-CP of sparse high-order interaction model (SHIM) which is sufficiently flexible as it can take into account high-order interactions among variables.
  arXiv  Detail & Related papers  (2022-05-28T03:23:56Z)
• Conformal prediction for the design problem [72.14982816083297]
  In many real-world deployments of machine learning, we use a prediction algorithm to choose what data to test next. In such settings, there is a distinct type of distribution shift between the training and test data. We introduce a method to quantify predictive uncertainty in such settings.
  arXiv  Detail & Related papers  (2022-02-08T02:59:12Z)
• Individual Calibration with Randomized Forecasting [116.2086707626651]
  We show that calibration for individual samples is possible in the regression setup if the predictions are randomized. We design a training objective to enforce individual calibration and use it to train randomized regression functions.
  arXiv  Detail & Related papers  (2020-06-18T05:53:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.