Distribution-informed Efficient Conformal Prediction for Full Ranking

• URL: http://arxiv.org/abs/2601.23128v1
• Date: Fri, 30 Jan 2026 16:16:44 GMT
• Title: Distribution-informed Efficient Conformal Prediction for Full Ranking
• Authors: Wenbo Liao, Huipeng Huang, Chen Jia, Huajun Xi, Hao Zeng, Hongxin Wei,
• Abstract summary: Quantifying uncertainty is critical for the safe deployment of ranking models in real-world applications.<n>Recent work offers a rigorous solution using conformal prediction in a full ranking scenario, which aims to construct prediction sets for the absolute ranks of test items.<n>We propose Distribution-informed Conformal Ranking (DCR), which produces efficient prediction sets by deriving the exact distribution of non-conformity scores.
• Score: 22.380815981596403
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Quantifying uncertainty is critical for the safe deployment of ranking models in real-world applications. Recent work offers a rigorous solution using conformal prediction in a full ranking scenario, which aims to construct prediction sets for the absolute ranks of test items based on the relative ranks of calibration items. However, relying on upper bounds of non-conformity scores renders the method overly conservative, resulting in substantially large prediction sets. To address this, we propose Distribution-informed Conformal Ranking (DCR), which produces efficient prediction sets by deriving the exact distribution of non-conformity scores. In particular, we find that the absolute ranks of calibration items follow Negative Hypergeometric distributions, conditional on their relative ranks. DCR thus uses the rank distribution to derive non-conformity score distribution and determine conformal thresholds. We provide theoretical guarantees that DCR achieves improved efficiency over the baseline while ensuring valid coverage under mild assumptions. Extensive experiments demonstrate the superiority of DCR, reducing average prediction set size by up to 36%, while maintaining valid coverage.

Related papers

• Distribution-informed Online Conformal Prediction [53.674678995825666]
  We propose Conformal Optimistic Prediction (COP), an online conformal prediction algorithm incorporating underlying data pattern into the update rule.<n>COP produces tighter prediction sets when predictable pattern exists, while retaining valid coverage guarantees even when estimates are inaccurate.<n>We prove that COP can achieve valid coverage and construct shorter prediction intervals than other baselines.
  arXiv  Detail & Related papers  (2025-12-08T17:51:49Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Distributed Conformal Prediction via Message Passing [33.306901198295016]
  Conformal Prediction (CP) offers a robust post-hoc calibration framework.<n>We propose two message-passing-based approaches for achieving reliable inference via CP.
  arXiv  Detail & Related papers  (2025-01-24T14:47:42Z)
• Conformal Prediction Sets with Improved Conditional Coverage using Trust Scores [52.92618442300405]
  It is impossible to achieve exact, distribution-free conditional coverage in finite samples.<n>We propose an alternative conformal prediction algorithm that targets coverage where it matters most.
  arXiv  Detail & Related papers  (2025-01-17T12:01:56Z)
• Provable Uncertainty Decomposition via Higher-Order Calibration [4.969075533165688]
  We give a principled method for decomposing the predictive uncertainty of a model into aleatoric and epistemic components.<n>Our method is based on the new notion of higher-order calibration.<n>We demonstrate through experiments that our method produces meaningful uncertainty decompositions for image classification.
  arXiv  Detail & Related papers  (2024-12-25T07:26:36Z)
• Provably Reliable Conformal Prediction Sets in the Presence of Data Poisoning [53.42244686183879]
  Conformal prediction provides model-agnostic and distribution-free uncertainty quantification.<n>Yet, conformal prediction is not reliable under poisoning attacks where adversaries manipulate both training and calibration data.<n>We propose reliable prediction sets (RPS): the first efficient method for constructing conformal prediction sets with provable reliability guarantees under poisoning.
  arXiv  Detail & Related papers  (2024-10-13T15:37:11Z)
• Probabilistic Conformal Prediction with Approximate Conditional Validity [81.30551968980143]
  We develop a new method for generating prediction sets that combines the flexibility of conformal methods with an estimate of the conditional distribution. Our method consistently outperforms existing approaches in terms of conditional coverage.
  arXiv  Detail & Related papers  (2024-07-01T20:44:48Z)
• U-Calibration: Forecasting for an Unknown Agent [29.3181385170725]
  We show that optimizing forecasts for a single scoring rule cannot guarantee low regret for all possible agents. We present a new metric for evaluating forecasts that we call U-calibration, equal to the maximal regret of the sequence of forecasts.
  arXiv  Detail & Related papers  (2023-06-30T23:05:26Z)
• Distribution-free uncertainty quantification for classification under label shift [105.27463615756733]
  We focus on uncertainty quantification (UQ) for classification problems via two avenues. We first argue that label shift hurts UQ, by showing degradation in coverage and calibration. We examine these techniques theoretically in a distribution-free framework and demonstrate their excellent practical performance.
  arXiv  Detail & Related papers  (2021-03-04T20:51:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.