Optimizing Probabilistic Conformal Prediction with Vectorized Non-Conformity Scores

• URL: http://arxiv.org/abs/2410.13735v1
• Date: Thu, 17 Oct 2024 16:37:03 GMT
• Title: Optimizing Probabilistic Conformal Prediction with Vectorized Non-Conformity Scores
• Authors: Minxing Zheng, Shixiang Zhu,
• Abstract summary: We propose a novel framework that enhances efficiency by first vectorizing the non-conformity scores with ranked samples and then optimizing the shape of the prediction set by varying the quantiles for samples at the same rank. Our method delivers valid coverage while producing discontinuous and more efficient prediction sets, making it particularly suited for high-stakes applications.
• Score: 6.059745771017814
• License:
• Abstract: Generative models have shown significant promise in critical domains such as medical diagnosis, autonomous driving, and climate science, where reliable decision-making hinges on accurate uncertainty quantification. While probabilistic conformal prediction (PCP) offers a powerful framework for this purpose, its coverage efficiency -- the size of the uncertainty set -- is limited when dealing with complex underlying distributions and a finite number of generated samples. In this paper, we propose a novel PCP framework that enhances efficiency by first vectorizing the non-conformity scores with ranked samples and then optimizing the shape of the prediction set by varying the quantiles for samples at the same rank. Our method delivers valid coverage while producing discontinuous and more efficient prediction sets, making it particularly suited for high-stakes applications. We demonstrate the effectiveness of our approach through experiments on both synthetic and real-world datasets.

Related papers

• Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filtering [55.15192437680943]
  Generative models lack rigorous statistical guarantees for their outputs. We propose a sequential conformal prediction method producing prediction sets that satisfy a rigorous statistical guarantee. This guarantee states that with high probability, the prediction sets contain at least one admissible (or valid) example.
  arXiv  Detail & Related papers  (2024-10-02T15:26:52Z)
• Probabilistic Conformal Prediction with Approximate Conditional Validity [81.30551968980143]
  We develop a new method for generating prediction sets that combines the flexibility of conformal methods with an estimate of the conditional distribution. Our method consistently outperforms existing approaches in terms of conditional coverage.
  arXiv  Detail & Related papers  (2024-07-01T20:44:48Z)
• The Penalized Inverse Probability Measure for Conformal Classification [0.5172964916120902]
  The work introduces the Penalized Inverse Probability (PIP) nonconformity score, and its regularized version RePIP, that allow the joint optimization of both efficiency and informativeness. The work shows how PIP-based conformal classifiers exhibit precisely the desired behavior in comparison with other nonconformity measures and strike a good balance between informativeness and efficiency.
  arXiv  Detail & Related papers  (2024-06-13T07:37:16Z)
• Stratified Prediction-Powered Inference for Hybrid Language Model Evaluation [62.2436697657307]
  Prediction-powered inference (PPI) is a method that improves statistical estimates based on limited human-labeled data. We propose a method called Stratified Prediction-Powered Inference (StratPPI) We show that the basic PPI estimates can be considerably improved by employing simple data stratification strategies.
  arXiv  Detail & Related papers  (2024-06-06T17:37:39Z)
• PAC-Bayes Generalization Certificates for Learned Inductive Conformal Prediction [27.434939269672288]
  We use PAC-Bayes theory to obtain generalization bounds on the coverage and the efficiency of set-valued predictors. We leverage these theoretical results to provide a practical algorithm for using calibration data to fine-tune the parameters of a model and score function. We evaluate the approach on regression and classification tasks, and outperform baselines calibrated using a Hoeffding bound-based PAC guarantee on ICP.
  arXiv  Detail & Related papers  (2023-12-07T19:40:44Z)
• Federated Conformal Predictors for Distributed Uncertainty Quantification [83.50609351513886]
  Conformal prediction is emerging as a popular paradigm for providing rigorous uncertainty quantification in machine learning. In this paper, we extend conformal prediction to the federated learning setting. We propose a weaker notion of partial exchangeability, better suited to the FL setting, and use it to develop the Federated Conformal Prediction framework.
  arXiv  Detail & Related papers  (2023-05-27T19:57:27Z)
• Variational Inference with Coverage Guarantees in Simulation-Based Inference [18.818573945984873]
  We propose Conformalized Amortized Neural Variational Inference (CANVI) CANVI constructs conformalized predictors based on each candidate, compares the predictors using a metric known as predictive efficiency, and returns the most efficient predictor. We prove lower bounds on the predictive efficiency of the regions produced by CANVI and explore how the quality of a posterior approximation relates to the predictive efficiency of prediction regions based on that approximation.
  arXiv  Detail & Related papers  (2023-05-23T17:24:04Z)
• Probabilistic Conformal Prediction Using Conditional Random Samples [73.26753677005331]
  PCP is a predictive inference algorithm that estimates a target variable by a discontinuous predictive set. It is efficient and compatible with either explicit or implicit conditional generative models.
  arXiv  Detail & Related papers  (2022-06-14T03:58:03Z)
• Sequential Domain Adaptation by Synthesizing Distributionally Robust Experts [14.656957226255628]
  Supervised domain adaptation aims to improve the predictive accuracy by exploiting additional labeled training samples from a source distribution close to the target distribution. We use the Bernstein online aggregation algorithm on the proposed family of robust experts to generate predictions for the sequential stream of target samples.
  arXiv  Detail & Related papers  (2021-06-01T08:51:55Z)
• Efficient Ensemble Model Generation for Uncertainty Estimation with Bayesian Approximation in Segmentation [74.06904875527556]
  We propose a generic and efficient segmentation framework to construct ensemble segmentation models. In the proposed method, ensemble models can be efficiently generated by using the layer selection method. We also devise a new pixel-wise uncertainty loss, which improves the predictive performance.
  arXiv  Detail & Related papers  (2020-05-21T16:08:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.