Reliable Conformal Prediction for Ordinal Classification Using the Ranked Probability Score

• URL: http://arxiv.org/abs/2606.24959v1
• Date: Tue, 23 Jun 2026 08:55:13 GMT
• Title: Reliable Conformal Prediction for Ordinal Classification Using the Ranked Probability Score
• Abstract summary: Conformal prediction (CP) provides distribution-free prediction sets with marginal coverage guarantees.<n>We introduce a CP method for ordinal classification based on the ranked probability score (RPS)<n>RPS yields median-centered contiguous prediction sets by construction.
• Score: 26.03725296495402
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Ordinal classification (OC) arises in high-stakes domains such as medicine and finance, where uncertainty quantification must account for the severity of ordinal errors. Conformal prediction (CP) provides distribution-free prediction sets with marginal coverage guarantees; however, its practical effectiveness depends critically on the choice of nonconformity function. We introduce a CP method for ordinal classification based on the ranked probability score (RPS), a proper scoring rule defined over cumulative predictive distributions. Although it reflects ordinal risk quite naturally, it has largely been neglected in conformal ordinal prediction (COP). When used as a measure of nonconformity, RPS yields median-centered contiguous prediction sets by construction. The method is model-agnostic, supports both assessed and grouped ordered categorical outcomes, and permits efficient implementation compared to greedy interval selection procedures. Across multiple ordinal image and tabular datasets, RPS-based CP produces contiguous prediction sets and strikes a favorable balance between prediction set width and the magnitude of ordinal miscoverage relative to existing CP methods.

Related papers

• Distribution-informed Efficient Conformal Prediction for Full Ranking [22.380815981596403]
  Quantifying uncertainty is critical for the safe deployment of ranking models in real-world applications.<n>Recent work offers a rigorous solution using conformal prediction in a full ranking scenario, which aims to construct prediction sets for the absolute ranks of test items.<n>We propose Distribution-informed Conformal Ranking (DCR), which produces efficient prediction sets by deriving the exact distribution of non-conformity scores.
  arXiv  Detail & Related papers  (2026-01-30T16:16:44Z)
• Distribution-informed Online Conformal Prediction [53.674678995825666]
  We propose Conformal Optimistic Prediction (COP), an online conformal prediction algorithm incorporating underlying data pattern into the update rule.<n>COP produces tighter prediction sets when predictable pattern exists, while retaining valid coverage guarantees even when estimates are inaccurate.<n>We prove that COP can achieve valid coverage and construct shorter prediction intervals than other baselines.
  arXiv  Detail & Related papers  (2025-12-08T17:51:49Z)
• Provably Minimum-Length Conformal Prediction Sets for Ordinal Classification [17.822517655243427]
  We propose an ordinal-CP method that is model-agnostic and provides instance-level optimal prediction intervals.<n>Experiments on four benchmark datasets from diverse domains are conducted to demonstrate the significantly improved predictive efficiency.
  arXiv  Detail & Related papers  (2025-11-20T23:00:15Z)
• Optimal Conformal Prediction under Epistemic Uncertainty [61.46247583794497]
  Conformal prediction (CP) is a popular framework for representing uncertainty.<n>We introduce Bernoulli prediction sets (BPS) which produce the smallest prediction sets that ensure conditional coverage.<n>When given first-order predictions, BPS reduces to the well-known adaptive prediction sets (APS)
  arXiv  Detail & Related papers  (2025-05-25T08:32:44Z)
• Conformal Prediction Sets with Improved Conditional Coverage using Trust Scores [52.92618442300405]
  It is impossible to achieve exact, distribution-free conditional coverage in finite samples.<n>We propose an alternative conformal prediction algorithm that targets coverage where it matters most.
  arXiv  Detail & Related papers  (2025-01-17T12:01:56Z)
• Posterior Conformal Prediction [11.576645675675096]
  Conformal prediction is a popular technique for constructing prediction intervals with distribution-free coverage guarantees. This article introduces a new method, posterior conformal prediction (PCP), which generates prediction intervals with both marginal and approximate conditional validity. We evaluate the performance of PCP on diverse datasets from socio-economic, scientific and healthcare applications.
  arXiv  Detail & Related papers  (2024-09-29T14:09:07Z)
• Probabilistic Conformal Prediction with Approximate Conditional Validity [81.30551968980143]
  We develop a new method for generating prediction sets that combines the flexibility of conformal methods with an estimate of the conditional distribution. Our method consistently outperforms existing approaches in terms of conditional coverage.
  arXiv  Detail & Related papers  (2024-07-01T20:44:48Z)
• Distribution-free Conformal Prediction for Ordinal Classification [0.0]
  Ordinal classification is common in real applications where the target variable has natural ordering among the class labels. New conformal prediction methods are developed for constructing contiguous and non-contiguous prediction sets.
  arXiv  Detail & Related papers  (2024-04-25T13:49:59Z)
• Likelihood Ratio Confidence Sets for Sequential Decision Making [51.66638486226482]
  We revisit the likelihood-based inference principle and propose to use likelihood ratios to construct valid confidence sequences. Our method is especially suitable for problems with well-specified likelihoods. We show how to provably choose the best sequence of estimators and shed light on connections to online convex optimization.
  arXiv  Detail & Related papers  (2023-11-08T00:10:21Z)
• Conformal prediction set for time-series [16.38369532102931]
  Uncertainty quantification is essential to studying complex machine learning methods. We develop Ensemble Regularized Adaptive Prediction Set (ERAPS) to construct prediction sets for time-series. We show valid marginal and conditional coverage by ERAPS, which also tends to yield smaller prediction sets than competing methods.
  arXiv  Detail & Related papers  (2022-06-15T23:48:53Z)
• Selective Classification via One-Sided Prediction [54.05407231648068]
  One-sided prediction (OSP) based relaxation yields an SC scheme that attains near-optimal coverage in the practically relevant high target accuracy regime. We theoretically derive bounds generalization for SC and OSP, and empirically we show that our scheme strongly outperforms state of the art methods in coverage at small error levels.
  arXiv  Detail & Related papers  (2020-10-15T16:14:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.