Online Selective Conformal Prediction with Asymmetric Rules: A Permutation Test Approach

• URL: http://arxiv.org/abs/2602.10018v1
• Date: Tue, 10 Feb 2026 17:39:36 GMT
• Title: Online Selective Conformal Prediction with Asymmetric Rules: A Permutation Test Approach
• Authors: Mingyi Zheng, Ying Jin,
• Abstract summary: Selective conformal prediction aims to construct prediction sets with valid coverage for a test unit conditional on it being selected by a data-driven mechanism.<n>Existing methods only address a limited collection of selection mechanisms.<n>We propose PErmutation-based Mondrian Conformal Inference (PEMI) for selective conformal prediction with arbitrary asymmetric selection rules.
• Score: 9.317702091531174
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Selective conformal prediction aims to construct prediction sets with valid coverage for a test unit conditional on it being selected by a data-driven mechanism. While existing methods in the offline setting handle any selection mechanism that is permutation invariant to the labeled data, their extension to the online setting -- where data arrives sequentially and later decisions depend on earlier ones -- is challenged by the fact that the selection mechanism is naturally asymmetric. As such, existing methods only address a limited collection of selection mechanisms. In this paper, we propose PErmutation-based Mondrian Conformal Inference (PEMI), a general permutation-based framework for selective conformal prediction with arbitrary asymmetric selection rules. Motivated by full and Mondrian conformal prediction, PEMI identifies all permutations of the observed data (or a Monte-Carlo subset thereof) that lead to the same selection event, and calibrates a prediction set using conformity scores over this selection-preserving reference set. Under standard exchangeability conditions, our prediction sets achieve finite-sample exact selection-conditional coverage for any asymmetric selection mechanism and any prediction model. PEMI naturally incorporates additional offline labeled data, extends to selection mechanisms with multiple test samples, and achieves FCR control with fine-grained selection taxonomies. We further work out several efficient instantiations for commonly-used online selection rules, including covariate-based rules, conformal p/e-values-based procedures, and selection based on earlier outcomes. Finally, we demonstrate the efficacy of our methods across various selection rules on a real drug discovery dataset and investigate their performance via simulations.

Related papers

• LEC: Linear Expectation Constraints for False-Discovery Control in Selective Prediction and Routing Systems [95.35293543918762]
  Large language models (LLMs) often generate unreliable answers, while uncertainty methods fail to fully distinguish correct from incorrect predictions.<n>We address this issue through the lens of false discovery rate (FDR) control, ensuring that among all accepted predictions, the proportion of errors does not exceed a target risk level.<n>We propose LEC, which reinterprets selective prediction as a constrained decision problem by enforcing a Linear Expectation Constraint.
  arXiv  Detail & Related papers  (2025-12-01T11:27:09Z)
• Online Conformal Selection with Accept-to-Reject Changes [14.619101485265322]
  Online Conformal Selection with Accept-to-Reject Changes (dubbed OCS-ARC) is proposed.<n>It incorporates online Benjamini-Hochberg procedure into the candidate selection process.<n>We provide theoretical guarantees that OCS-ARC controls the false discovery rate (FDR) at or below the nominal level at any timestep.
  arXiv  Detail & Related papers  (2025-08-19T13:58:38Z)
• Valid Selection among Conformal Sets [53.016786692105796]
  Conformal prediction offers a distribution-free framework for constructing prediction sets with coverage guarantees.<n> selecting the most desirable set, such as the smallest, can invalidate the coverage guarantees.<n>We propose a stability-based approach that ensures coverage for the selected prediction set.
  arXiv  Detail & Related papers  (2025-06-25T06:59:55Z)
• A Principled Approach to Randomized Selection under Uncertainty: Applications to Peer Review and Grant Funding [61.86327960322782]
  We propose a principled framework for randomized decision-making based on interval estimates of the quality of each item.<n>We introduce MERIT, an optimization-based method that maximizes the worst-case expected number of top candidates selected.<n>We prove that MERIT satisfies desirable axiomatic properties not guaranteed by existing approaches.
  arXiv  Detail & Related papers  (2025-06-23T19:59:30Z)
• Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filtering [55.15192437680943]
  Generative models lack rigorous statistical guarantees for their outputs.<n>We propose a sequential conformal prediction method producing prediction sets that satisfy a rigorous statistical guarantee.<n>This guarantee states that with high probability, the prediction sets contain at least one admissible (or valid) example.
  arXiv  Detail & Related papers  (2024-10-02T15:26:52Z)
• Conformal online model aggregation [36.191356601153146]
  Conformal prediction equips machine learning models with a reasonable notion of uncertainty quantification without making strong distributional assumptions.<n>We propose a wrapper that takes in several conformal prediction sets and outputs a single adaptively-combined prediction set.<n>Our method, called conformal online model aggregation (COMA), is based on combining the prediction sets from several algorithms by weighted voting.
  arXiv  Detail & Related papers  (2024-03-22T15:40:06Z)
• CAP: A General Algorithm for Online Selective Conformal Prediction with FCR Control [4.137346786534721]
  It is important to control the real-time false coverage-statement rate (FCR) which measures the overall miscoverage level.<n>We develop a general framework named CAP that performs an adaptive pick rule on historical data to construct a calibration set.<n>We prove that CAP can achieve an exact selection-conditional coverage guarantee in the finite-sample and distribution-free regimes.
  arXiv  Detail & Related papers  (2024-03-12T15:07:20Z)
• Confidence on the Focal: Conformal Prediction with Selection-Conditional Coverage [6.010965256037659]
  Conformal prediction builds marginally valid prediction intervals that cover the unknown outcome of a randomly drawn test point.<n>In practice, data-driven methods are often used to identify specific test unit(s) of interest.<n>This paper presents a general framework for constructing a prediction set with finite-sample exact coverage.
  arXiv  Detail & Related papers  (2024-03-06T17:18:24Z)
• Large Language Models Are Not Robust Multiple Choice Selectors [117.72712117510953]
  Multiple choice questions (MCQs) serve as a common yet important task format in the evaluation of large language models (LLMs) This work shows that modern LLMs are vulnerable to option position changes due to their inherent "selection bias" We propose a label-free, inference-time debiasing method, called PriDe, which separates the model's prior bias for option IDs from the overall prediction distribution.
  arXiv  Detail & Related papers  (2023-09-07T17:44:56Z)
• Conformal Language Modeling [61.94417935386489]
  We propose a novel approach to conformal prediction for generative language models (LMs) Standard conformal prediction produces prediction sets with rigorous, statistical guarantees. We demonstrate the promise of our approach on multiple tasks in open-domain question answering, text summarization, and radiology report generation.
  arXiv  Detail & Related papers  (2023-06-16T21:55:08Z)
• On the Validity of Conformal Prediction for Network Data Under Non-Uniform Sampling [2.0305676256390934]
  We study the properties of conformal prediction for network data under various sampling mechanisms. We show that the sampled subarray is exchangeable conditional on the selection event if the selection rule satisfies a permutation invariance property. Our result implies the finite-sample validity of conformal prediction for certain selection events related to ego networks and snowball sampling.
  arXiv  Detail & Related papers  (2023-06-12T17:21:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.