Beyond Conformal Predictors: Adaptive Conformal Inference with Confidence Predictors

• URL: http://arxiv.org/abs/2409.15548v4
• Date: Thu, 26 Jun 2025 20:25:03 GMT
• Title: Beyond Conformal Predictors: Adaptive Conformal Inference with Confidence Predictors
• Authors: Johan Hallberg Szabadváry, Tuwe Löfström,
• Abstract summary: This study shows that the desirable properties of Adaptive Conformal Inference (ACI) do not require the use of Conformal Predictors (CP)<n>We empirically investigate the performance of Non-Conformal Confidence Predictors (NCCP) against CP when used with ACI on non-exchangeable data.
• Score: 1.3812010983144802
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Adaptive Conformal Inference (ACI) provides finite-sample coverage guarantees, enhancing the prediction reliability under non-exchangeability. This study demonstrates that these desirable properties of ACI do not require the use of Conformal Predictors (CP). We show that the guarantees hold for the broader class of confidence predictors, defined by the requirement of producing nested prediction sets, a property we argue is essential for meaningful confidence statements. We empirically investigate the performance of Non-Conformal Confidence Predictors (NCCP) against CP when used with ACI on non-exchangeable data. In online settings, the NCCP offers significant computational advantages while maintaining a comparable predictive efficiency. In batch settings, inductive NCCP (INCCP) can outperform inductive CP (ICP) by utilising the full training dataset without requiring a separate calibration set, leading to improved efficiency, particularly when the data are limited. Although these initial results highlight NCCP as a theoretically sound and practically effective alternative to CP for uncertainty quantification with ACI in non-exchangeable scenarios, further empirical studies are warranted across diverse datasets and predictors.

Related papers

• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• OT Score: An OT based Confidence Score for Unsupervised Domain Adaptation [4.544755512744677]
  We introduce the Optimal Transport (OT) score, a confidence metric derived from a novel theoretical analysis.<n>We show that OT score significantly outperforms existing confidence metrics across diverse adaptation scenarios.
  arXiv  Detail & Related papers  (2025-05-16T20:09:05Z)
• SConU: Selective Conformal Uncertainty in Large Language Models [59.25881667640868]
  We propose a novel approach termed Selective Conformal Uncertainty (SConU) We develop two conformal p-values that are instrumental in determining whether a given sample deviates from the uncertainty distribution of the calibration set at a specific manageable risk level. Our approach not only facilitates rigorous management of miscoverage rates across both single-domain and interdisciplinary contexts, but also enhances the efficiency of predictions.
  arXiv  Detail & Related papers  (2025-04-19T03:01:45Z)
• E-Values Expand the Scope of Conformal Prediction [49.1574468325115]
  Conformal prediction is a powerful framework for distribution-free uncertainty quantification. In this paper, we explore an alternative approach based on e-values, known as conformal e-prediction. E-values offer key advantages that cannot be achieved with p-values, enabling new theoretical and practical capabilities.
  arXiv  Detail & Related papers  (2025-03-17T10:54:30Z)
• Conformal Prediction Sets with Improved Conditional Coverage using Trust Scores [52.92618442300405]
  It is impossible to achieve exact, distribution-free conditional coverage in finite samples. We propose an alternative conformal prediction algorithm that targets coverage where it matters most.
  arXiv  Detail & Related papers  (2025-01-17T12:01:56Z)
• On the Role of Surrogates in Conformal Inference of Individual Causal Effects [0.0]
  We introduce underlineSurrogate-assisted underlineConformal underlineInference for underlineEfficient IunderlineNdividual underlineCausal underlineEffects (SCIENCE)<n>SCIENCE is a framework designed to construct more efficient prediction intervals for individual treatment effects (ITEs)<n>It is applied to the phase 3 Moderna COVE COVID-19 vaccine trial.
  arXiv  Detail & Related papers  (2024-12-16T21:36:11Z)
• Generative Conformal Prediction with Vectorized Non-Conformity Scores [6.059745771017814]
  Conformal prediction provides model-agnostic uncertainty quantification with guaranteed coverage. We propose a generative conformal prediction framework with vectorized non-conformity scores. We construct adaptive uncertainty sets using density-ranked uncertainty balls.
  arXiv  Detail & Related papers  (2024-10-17T16:37:03Z)
• Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning [53.42244686183879]
  Conformal prediction provides model-agnostic and distribution-free uncertainty quantification. Yet, conformal prediction is not reliable under poisoning attacks where adversaries manipulate both training and calibration data. We propose reliable prediction sets (RPS): the first efficient method for constructing conformal prediction sets with provable reliability guarantees under poisoning.
  arXiv  Detail & Related papers  (2024-10-13T15:37:11Z)
• Online scalable Gaussian processes with conformal prediction for guaranteed coverage [32.21093722162573]
  The consistency of the resulting uncertainty values hinges on the premise that the learning function conforms to the properties specified by the GP model. We propose to wed the GP with the prevailing conformal prediction (CP), a distribution-free post-processing framework that produces it prediction sets with a provably valid coverage.
  arXiv  Detail & Related papers  (2024-10-07T19:22:15Z)
• Calibrated Probabilistic Forecasts for Arbitrary Sequences [58.54729945445505]
  Real-world data streams can change unpredictably due to distribution shifts, feedback loops and adversarial actors. We present a forecasting framework ensuring valid uncertainty estimates regardless of how data evolves.
  arXiv  Detail & Related papers  (2024-09-27T21:46:42Z)
• Probabilistic Conformal Prediction with Approximate Conditional Validity [81.30551968980143]
  We develop a new method for generating prediction sets that combines the flexibility of conformal methods with an estimate of the conditional distribution. Our method consistently outperforms existing approaches in terms of conditional coverage.
  arXiv  Detail & Related papers  (2024-07-01T20:44:48Z)
• The Penalized Inverse Probability Measure for Conformal Classification [0.5172964916120902]
  The work introduces the Penalized Inverse Probability (PIP) nonconformity score, and its regularized version RePIP, that allow the joint optimization of both efficiency and informativeness. The work shows how PIP-based conformal classifiers exhibit precisely the desired behavior in comparison with other nonconformity measures and strike a good balance between informativeness and efficiency.
  arXiv  Detail & Related papers  (2024-06-13T07:37:16Z)
• Robust Conformal Prediction Using Privileged Information [17.886554223172517]
  We develop a method to generate prediction sets with a guaranteed coverage rate that is robust to corruptions in the training data. Our approach builds on conformal prediction, a powerful framework to construct prediction sets that are valid under the i.i.d assumption.
  arXiv  Detail & Related papers  (2024-06-08T08:56:47Z)
• Conformal Prediction with Learned Features [22.733758606168873]
  We propose Partition Learning Conformal Prediction (PLCP) to improve conditional validity of prediction sets. We implement PLCP efficiently with gradient alternating descent, utilizing off-the-shelf machine learning models. Our experimental results over four real-world and synthetic datasets show the superior performance of PLCP.
  arXiv  Detail & Related papers  (2024-04-26T15:43:06Z)
• Confidence-aware Fine-tuning of Sequential Recommendation Systems via Conformal Prediction [46.76846936581471]
  In Sequential Recommendation Systems (SRecsys), traditional training approaches that rely on Cross-Entropy (CE) loss often prioritize accuracy but fail to align well with user satisfaction metrics.<n>We propose textbfCPFT, a novel fine-tuning framework that integrates Conformal Prediction (CP)-based losses with CE loss to optimize accuracy alongside confidence that better aligns with widely used top-$K$ metrics.
  arXiv  Detail & Related papers  (2024-02-14T06:43:02Z)
• Efficient Conformal Prediction under Data Heterogeneity [79.35418041861327]
  Conformal Prediction (CP) stands out as a robust framework for uncertainty quantification. Existing approaches for tackling non-exchangeability lead to methods that are not computable beyond the simplest examples. This work introduces a new efficient approach to CP that produces provably valid confidence sets for fairly general non-exchangeable data distributions.
  arXiv  Detail & Related papers  (2023-12-25T20:02:51Z)
• PAC-Bayes Generalization Certificates for Learned Inductive Conformal Prediction [27.434939269672288]
  We use PAC-Bayes theory to obtain generalization bounds on the coverage and the efficiency of set-valued predictors. We leverage these theoretical results to provide a practical algorithm for using calibration data to fine-tune the parameters of a model and score function. We evaluate the approach on regression and classification tasks, and outperform baselines calibrated using a Hoeffding bound-based PAC guarantee on ICP.
  arXiv  Detail & Related papers  (2023-12-07T19:40:44Z)
• Conformal Prediction for Deep Classifier via Label Ranking [29.784336674173616]
  Conformal prediction is a statistical framework that generates prediction sets with a desired coverage guarantee. We propose a novel algorithm named $textitSorted Adaptive Prediction Sets$ (SAPS) SAPS discards all the probability values except for the maximum softmax probability.
  arXiv  Detail & Related papers  (2023-10-10T08:54:14Z)
• Federated Conformal Predictors for Distributed Uncertainty Quantification [83.50609351513886]
  Conformal prediction is emerging as a popular paradigm for providing rigorous uncertainty quantification in machine learning. In this paper, we extend conformal prediction to the federated learning setting. We propose a weaker notion of partial exchangeability, better suited to the FL setting, and use it to develop the Federated Conformal Prediction framework.
  arXiv  Detail & Related papers  (2023-05-27T19:57:27Z)
• Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
  We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores. We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
  arXiv  Detail & Related papers  (2023-02-23T18:57:14Z)
• Learning Optimal Conformal Classifiers [32.68483191509137]
  Conformal prediction (CP) is used to predict confidence sets containing the true class with a user-specified probability. This paper explores strategies to differentiate through CP during training with the goal of training model with the conformal wrapper end-to-end. We show that conformal training (ConfTr) outperforms state-of-the-art CP methods for classification by reducing the average confidence set size.
  arXiv  Detail & Related papers  (2021-10-18T11:25:33Z)
• Optimized conformal classification using gradient descent approximation [0.2538209532048866]
  Conformal predictors allow predictions to be made with a user-defined confidence level. We consider an approach to train the conformal predictor directly with maximum predictive efficiency. We test the method on several real world data sets and find that the method is promising.
  arXiv  Detail & Related papers  (2021-05-24T13:14:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.