Conformal Prediction with Macro-Coverage Guarantees

• URL: http://arxiv.org/abs/2606.28598v1
• Date: Fri, 26 Jun 2026 20:47:33 GMT
• Title: Conformal Prediction with Macro-Coverage Guarantees
• Abstract summary: We show that label-weighted conformal prediction can be used to produce prediction sets with a finite-sample macro-coverage guarantee.<n>We further characterize the form of the smallest prediction sets satisfying a given generalized macro-coverage objective.
• Score: 4.6071451559137175
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Prediction sets should have high coverage to be useful, but some coverage notions are more practically relevant than others. In the classification setting, class-conditional coverage requires that the prediction set (i.e., the set of candidate labels for a new test point) must achieve the target accuracy level within each class, which may be challenging to satisfy when many classes are rare and have few calibration points. At the other extreme, marginal coverage requires only that coverage holds on average over the distribution of all classes, which can lead to low-probability labels being essentially ignored. To find a middle ground, recent work has introduced macro-coverage, defined as the unweighted average of class-conditional coverages. Macro-coverage offers a compromise between marginal coverage and class-conditional coverage that is particularly appropriate for long-tailed settings. In this work, we show that label-weighted conformal prediction can be used to produce prediction sets with a finite-sample macro-coverage guarantee, and more generally a guarantee on a family of generalized macro-coverage objectives that aggregate coverage at the level of arbitrary class groupings and take a weighted average. We further characterize the form of the smallest prediction sets satisfying a given generalized macro-coverage objective and propose a corresponding conformal score function. We validate our theoretical results on two large-scale image classification datasets.

Related papers

• Conditional Coverage Diagnostics for Conformal Prediction [47.93989136542648]
  We show that conditional coverage estimation can be a classification problem.<n>We call the resulting family of metrics excess risk of the target coverage (ERT)<n>We release an open-source package for ERT as well as previous conditional coverage metrics.
  arXiv  Detail & Related papers  (2025-12-12T18:47:39Z)
• Adaptive Coverage Policies in Conformal Prediction [46.298122008420414]
  We leverage advances in e-values and post-hoc conformal inference to improve conformal prediction methods.<n>We propose to optimize an adaptive coverage policy by training a neural network using a leave-one-out procedure on a calibration set.
  arXiv  Detail & Related papers  (2025-10-05T18:37:22Z)
• Conformal Prediction for Long-Tailed Classification [4.343271732951346]
  Many real-world classification problems, such as plant identification, have extremely long-tailed class distributions.<n>Existing conformal prediction methods force practitioners to make a binary choice between small sets with poor class-conditional coverage or sets with very good class-conditional coverage.<n>We propose methods with guaranteed marginal coverage that smoothly trade off between set size and class-conditional coverage.
  arXiv  Detail & Related papers  (2025-07-09T14:08:50Z)
• Conformal Prediction Sets with Improved Conditional Coverage using Trust Scores [52.92618442300405]
  It is impossible to achieve exact, distribution-free conditional coverage in finite samples.<n>We propose an alternative conformal prediction algorithm that targets coverage where it matters most.
  arXiv  Detail & Related papers  (2025-01-17T12:01:56Z)
• Conformal Prediction for Class-wise Coverage via Augmented Label Rank Calibration [30.039997048267]
  This paper proposes the Rank Calibrated Class-conditional CP (RC3P) algorithm to reduce the prediction set sizes to achieve class-conditional coverage.<n>Experiments on multiple real-world datasets demonstrate that RC3P achieves class-wise coverage and 26.25% reduction in prediction set sizes on average.
  arXiv  Detail & Related papers  (2024-06-10T22:01:34Z)
• Equal Opportunity of Coverage in Fair Regression [50.76908018786335]
  We study fair machine learning (ML) under predictive uncertainty to enable reliable and trustworthy decision-making. We propose Equal Opportunity of Coverage (EOC) that aims to achieve two properties: (1) coverage rates for different groups with similar outcomes are close, and (2) the coverage rate for the entire population remains at a predetermined level.
  arXiv  Detail & Related papers  (2023-11-03T21:19:59Z)
• Class-Conditional Conformal Prediction with Many Classes [60.8189977620604]
  We propose a method called clustered conformal prediction that clusters together classes having "similar" conformal scores. We find that clustered conformal typically outperforms existing methods in terms of class-conditional coverage and set size metrics.
  arXiv  Detail & Related papers  (2023-06-15T17:59:02Z)
• Practical Adversarial Multivalid Conformal Prediction [27.179891682629183]
  We give a generic conformal prediction method for sequential prediction. It achieves target empirical coverage guarantees against adversarially chosen data. It is computationally lightweight -- comparable to split conformal prediction.
  arXiv  Detail & Related papers  (2022-06-02T14:33:00Z)
• Cautious Active Clustering [79.23797234241471]
  We consider the problem of classification of points sampled from an unknown probability measure on a Euclidean space. Our approach is to consider the unknown probability measure as a convex combination of the conditional probabilities for each class.
  arXiv  Detail & Related papers  (2020-08-03T23:47:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.