Robust Conformalized Selection with Noisy Responses

• URL: http://arxiv.org/abs/2607.22985v1
• Date: Sat, 25 Jul 2026 01:51:46 GMT
• Title: Robust Conformalized Selection with Noisy Responses
• Abstract summary: Conformalized selection has been widely applied to select high-quality candidates from large datasets.<n>We formulate the above tasks as selecting candidates with true predicted labels or with responses exceeding certain values.<n>We propose Robust Conformalized Selection (RCS), a unified framework for selective classification with valid FDR control under general label contamination.
• Score: 23.306273801765297
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Conformalized selection has been widely applied to select high-quality candidates from large datasets with rigorous uncertainty quantification, such as reliable labeling, drug discovery, and the alignment of large language models. Nevertheless, existing methods assume clean responses on calibration data, an assumption that rarely holds in practice. In this paper, we formulate the above tasks as selecting candidates with true predicted labels or with responses exceeding certain values. We demonstrate that existing conformal selection methods fail to control the false discovery rate (FDR) or suffer from severe power loss under contaminated calibration data. To that end, we propose Robust Conformalized Selection (RCS), a unified framework for selective classification with valid FDR control under general label contamination. The key insight of RCS lies in a novel statistical reduction: by separately conditioning on different classes, we translate the intractable label noise into a localized covariate shift problem, which then enables a covariate-adjusted empirical-Bayes-type estimate of the number of false selections. Statistical properties such as the asymptotic FDR control, power optimality, and robustness of RCS are established. We further develop an instantiation of RCS under randomized response model, and also apply RCS to the task of selecting candidates with large response values. Extensive experiments on both simulated and real-world datasets demonstrate the effectiveness of RCS.

Related papers

• Beyond Fixed False Discovery Rates: Post-Hoc Conformal Selection with E-Variables [31.204606769083068]
  Conformal selection (CS) uses calibration data to identify test inputs whose unobserved outcomes are likely to satisfy a pre-specified minimal quality requirement, while controlling the false discovery rate (FDR)<n>Existing methods fix the target FDR level before observing data, which prevents the user from adapting the balance between number of selected test inputs and FDR to downstream needs and constraints based on the available data.<n>We introduce post-hoc CS (PH-CS), which generates a path of candidate selection sets, each paired with a data-driven false discovery proportion (FDP) estimate.
  arXiv  Detail & Related papers  (2026-04-13T11:02:44Z)
• LEC: Linear Expectation Constraints for False-Discovery Control in Selective Prediction and Routing Systems [95.35293543918762]
  Large language models (LLMs) often generate unreliable answers, while uncertainty methods fail to fully distinguish correct from incorrect predictions.<n>We address this issue through the lens of false discovery rate (FDR) control, ensuring that among all accepted predictions, the proportion of errors does not exceed a target risk level.<n>We propose LEC, which reinterprets selective prediction as a constrained decision problem by enforcing a Linear Expectation Constraint.
  arXiv  Detail & Related papers  (2025-12-01T11:27:09Z)
• SAFER: Risk-Constrained Sample-then-Filter in Large Language Models [38.97678256807034]
  We introduce a two-stage risk control framework comprising abstention-aware sampling and conformalized filtering.<n>We show that SAFER is compatible with various task-specific admission criteria and calibration-test split ratios.
  arXiv  Detail & Related papers  (2025-10-11T12:12:41Z)
• Reference-Free Rating of LLM Responses via Latent Information [53.463883683503106]
  We study the common practice of asking a judge model to assign Likert-scale scores to free-text responses.<n>We then propose and evaluate Latent Judges, which derive scalar ratings from internal model signals.<n>Across a broad suite of pairwise and single-rating benchmarks, latent methods match or surpass standard prompting.
  arXiv  Detail & Related papers  (2025-09-29T12:15:52Z)
• Online Conformal Selection with Accept-to-Reject Changes [14.619101485265322]
  Online Conformal Selection with Accept-to-Reject Changes (dubbed OCS-ARC) is proposed.<n>It incorporates online Benjamini-Hochberg procedure into the candidate selection process.<n>We provide theoretical guarantees that OCS-ARC controls the false discovery rate (FDR) at or below the nominal level at any timestep.
  arXiv  Detail & Related papers  (2025-08-19T13:58:38Z)
• COIN: Uncertainty-Guarding Selective Question Answering for Foundation Models with Provable Risk Guarantees [51.5976496056012]
  COIN is an uncertainty-guarding selection framework that calibrates statistically valid thresholds to filter a single generated answer per question.<n>COIN estimates the empirical error rate on a calibration set and applies confidence interval methods to establish a high-probability upper bound on the true error rate.<n>We demonstrate COIN's robustness in risk control, strong test-time power in retaining admissible answers, and predictive efficiency under limited calibration data.
  arXiv  Detail & Related papers  (2025-06-25T07:04:49Z)
• Multivariate Conformal Selection [9.431551477608528]
  We propose a generalization of Conformal Selection (CS) to provide rigorous uncertainty quantification.<n>We present two variants: mCS-dist, using distance-based scores, and mCS-learn, which learns optimal scores via differentiable optimization.<n>Experiments on simulated and real-world datasets demonstrate that mCS significantly improves selection power while maintaining False Discovery Rate (FDR) control.
  arXiv  Detail & Related papers  (2025-05-01T23:33:57Z)
• Noise-Adaptive Conformal Classification with Marginal Coverage [53.74125453366155]
  We introduce an adaptive conformal inference method capable of efficiently handling deviations from exchangeability caused by random label noise.<n>We validate our method through extensive numerical experiments demonstrating its effectiveness on synthetic and real data sets.
  arXiv  Detail & Related papers  (2025-01-29T23:55:23Z)
• Noisy Correspondence Learning with Self-Reinforcing Errors Mitigation [63.180725016463974]
  Cross-modal retrieval relies on well-matched large-scale datasets that are laborious in practice. We introduce a novel noisy correspondence learning framework, namely textbfSelf-textbfReinforcing textbfErrors textbfMitigation (SREM)
  arXiv  Detail & Related papers  (2023-12-27T09:03:43Z)
• Knockoffs-SPR: Clean Sample Selection in Learning with Noisy Labels [56.81761908354718]
  We propose a novel theoretically guaranteed clean sample selection framework for learning with noisy labels. Knockoffs-SPR can be regarded as a sample selection module for a standard supervised training pipeline. We further combine it with a semi-supervised algorithm to exploit the support of noisy data as unlabeled data.
  arXiv  Detail & Related papers  (2023-01-02T07:13:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.