Beyond Fixed False Discovery Rates: Post-Hoc Conformal Selection with E-Variables

• URL: http://arxiv.org/abs/2604.11305v2
• Date: Fri, 17 Apr 2026 11:33:47 GMT
• Title: Beyond Fixed False Discovery Rates: Post-Hoc Conformal Selection with E-Variables
• Abstract summary: Conformal selection (CS) uses calibration data to identify test inputs whose unobserved outcomes are likely to satisfy a pre-specified minimal quality requirement, while controlling the false discovery rate (FDR)<n>Existing methods fix the target FDR level before observing data, which prevents the user from adapting the balance between number of selected test inputs and FDR to downstream needs and constraints based on the available data.<n>We introduce post-hoc CS (PH-CS), which generates a path of candidate selection sets, each paired with a data-driven false discovery proportion (FDP) estimate.
• Score: 31.204606769083068
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conformal selection (CS) uses calibration data to identify test inputs whose unobserved outcomes are likely to satisfy a pre-specified minimal quality requirement, while controlling the false discovery rate (FDR). Existing methods fix the target FDR level before observing data, which prevents the user from adapting the balance between number of selected test inputs and FDR to downstream needs and constraints based on the available data. For example, in genomics or neuroimaging, researchers often inspect the distribution of test statistics, and decide how aggressively to pursue candidates based on observed evidence strength and available follow-up resources. To address this limitation, we introduce {post-hoc CS} (PH-CS), which generates a path of candidate selection sets, each paired with a data-driven false discovery proportion (FDP) estimate. PH-CS lets the user select any operating point on this path by maximizing a user-specified utility, arbitrarily balancing selection size and FDR. Building on conformal e-variables and the e-Benjamini-Hochberg (e-BH) procedure, PH-CS is proved to provide a finite-sample post-hoc reliability guarantee whereby the ratio between estimated FDP level and true FDP is, on average, upper bounded by $1$, so that the average estimated FDP is, to first order, a valid upper bound on the true FDR. PH-CS is extended to control quality defined in terms of a general risk. Experiments on synthetic and real-world datasets demonstrate that, unlike CS, PH-CS can consistently satisfy user-imposed utility constraints while producing reliable FDP estimates and maintaining competitive FDR control.

Related papers

• Conformal Changepoint Localization and Root Cause Analysis with Corrupted Observations [55.76952683833966]
  This paper proposes weighted CONCH (W-CONCH) and weighted CROC (W-CROC) to reduce confidence set size when data may be corrupted.<n>Experiments on image-based and real-world changepoint and root-cause benchmarks show that uncertainty-based weighting substantially reduces confidence set size.
  arXiv  Detail & Related papers  (2026-07-29T05:16:59Z)
• Robust Conformalized Selection with Noisy Responses [23.306273801765297]
  Conformalized selection has been widely applied to select high-quality candidates from large datasets.<n>We formulate the above tasks as selecting candidates with true predicted labels or with responses exceeding certain values.<n>We propose Robust Conformalized Selection (RCS), a unified framework for selective classification with valid FDR control under general label contamination.
  arXiv  Detail & Related papers  (2026-07-25T01:51:46Z)
• Denoised Conformal Alignment for Reliable Selection of Conditional Average Treatment Effect Predictions [10.945117945979566]
  We study reliable selection for black-box CATE predictors.<n>We construct doubly robust proxy errors from pseudo-outcomes.<n>We propose Denoised Conformal Alignment.
  arXiv  Detail & Related papers  (2026-07-03T10:00:28Z)
• Everywhere Valid Bounds on False Discovery Proportions in Conformal Inference [19.796377044066286]
  Conformal inference is often used to select test samples whose conformal p-values fall below a threshold.<n>The quality of such methods is often measured by the false discovery proportion (FDP), defined as the fraction of incorrect selections.<n>This paper establishes finite-sample, distribution-free upper bounds on the FDP that hold simultaneously over all possible rejection thresholds.
  arXiv  Detail & Related papers  (2026-05-20T05:24:39Z)
• LEC: Linear Expectation Constraints for False-Discovery Control in Selective Prediction and Routing Systems [95.35293543918762]
  Large language models (LLMs) often generate unreliable answers, while uncertainty methods fail to fully distinguish correct from incorrect predictions.<n>We address this issue through the lens of false discovery rate (FDR) control, ensuring that among all accepted predictions, the proportion of errors does not exceed a target risk level.<n>We propose LEC, which reinterprets selective prediction as a constrained decision problem by enforcing a Linear Expectation Constraint.
  arXiv  Detail & Related papers  (2025-12-01T11:27:09Z)
• SAFER: Risk-Constrained Sample-then-Filter in Large Language Models [38.97678256807034]
  We introduce a two-stage risk control framework comprising abstention-aware sampling and conformalized filtering.<n>We show that SAFER is compatible with various task-specific admission criteria and calibration-test split ratios.
  arXiv  Detail & Related papers  (2025-10-11T12:12:41Z)
• High-Power Training Data Identification with Provable Statistical Guarantees [21.78494059028984]
  We introduce Provable Training Data Identification (PTDI), a rigorous method that identifies a set of training data with strict false discovery rate (FDR) control.<n>This entire procedure enables the discovery of training data with provable, strict FDR control and significantly boosted power.
  arXiv  Detail & Related papers  (2025-10-10T06:28:36Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Certifiably Byzantine-Robust Federated Conformal Prediction [49.23374238798428]
  We introduce a novel framework Rob-FCP, which executes robust federated conformal prediction effectively countering malicious clients. We empirically demonstrate the robustness of Rob-FCP against diverse proportions of malicious clients under a variety of Byzantine attacks.
  arXiv  Detail & Related papers  (2024-06-04T04:43:30Z)
• Error-based Knockoffs Inference for Controlled Feature Selection [49.99321384855201]
  We propose an error-based knockoff inference method by integrating the knockoff features, the error-based feature importance statistics, and the stepdown procedure together. The proposed inference procedure does not require specifying a regression model and can handle feature selection with theoretical guarantees.
  arXiv  Detail & Related papers  (2022-03-09T01:55:59Z)
• Post-Contextual-Bandit Inference [57.88785630755165]
  Contextual bandit algorithms are increasingly replacing non-adaptive A/B tests in e-commerce, healthcare, and policymaking. They can both improve outcomes for study participants and increase the chance of identifying good or even best policies. To support credible inference on novel interventions at the end of the study, we still want to construct valid confidence intervals on average treatment effects, subgroup effects, or value of new policies.
  arXiv  Detail & Related papers  (2021-06-01T12:01:51Z)
• Sparse Feature Selection Makes Batch Reinforcement Learning More Sample Efficient [62.24615324523435]
  This paper provides a statistical analysis of high-dimensional batch Reinforcement Learning (RL) using sparse linear function approximation. When there is a large number of candidate features, our result sheds light on the fact that sparsity-aware methods can make batch RL more sample efficient.
  arXiv  Detail & Related papers  (2020-11-08T16:48:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.