Parametric Scaling Law of Tuning Bias in Conformal Prediction

• URL: http://arxiv.org/abs/2502.03023v1
• Date: Wed, 05 Feb 2025 09:26:47 GMT
• Title: Parametric Scaling Law of Tuning Bias in Conformal Prediction
• Authors: Hao Zeng, Kangdao Liu, Bingyi Jing, Hongxin Wei,
• Abstract summary: We find that the tuning bias - the coverage gap introduced by leveraging the same dataset for tuning and calibration - is negligible for simple parameter tuning in many conformal prediction methods. We establish a theoretical framework to quantify the tuning bias and provide rigorous proof for the scaling law of the tuning bias by deriving its upper bound.
• Score: 11.970092440023956
• License:
• Abstract: Conformal prediction is a popular framework of uncertainty quantification that constructs prediction sets with coverage guarantees. To uphold the exchangeability assumption, many conformal prediction methods necessitate an additional holdout set for parameter tuning. Yet, the impact of violating this principle on coverage remains underexplored, making it ambiguous in practical applications. In this work, we empirically find that the tuning bias - the coverage gap introduced by leveraging the same dataset for tuning and calibration, is negligible for simple parameter tuning in many conformal prediction methods. In particular, we observe the scaling law of the tuning bias: this bias increases with parameter space complexity and decreases with calibration set size. Formally, we establish a theoretical framework to quantify the tuning bias and provide rigorous proof for the scaling law of the tuning bias by deriving its upper bound. In the end, we discuss how to reduce the tuning bias, guided by the theories we developed.

Related papers

• Semi-Supervised Risk Control via Prediction-Powered Inference [14.890609936348277]
  Risk-controlling prediction sets (RCPS) is a tool for transforming the output of any machine learning model to design a predictive rule with rigorous error rate control. We introduce a semi-supervised calibration procedure that leverages unlabeled data to rigorously tune the hyper- parameter. Our procedure builds upon the prediction-powered inference framework, carefully tailoring it to risk-controlling tasks.
  arXiv  Detail & Related papers  (2024-12-15T13:00:23Z)
• Online scalable Gaussian processes with conformal prediction for guaranteed coverage [32.21093722162573]
  The consistency of the resulting uncertainty values hinges on the premise that the learning function conforms to the properties specified by the GP model. We propose to wed the GP with the prevailing conformal prediction (CP), a distribution-free post-processing framework that produces it prediction sets with a provably valid coverage.
  arXiv  Detail & Related papers  (2024-10-07T19:22:15Z)
• Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filtering [55.15192437680943]
  Generative models lack rigorous statistical guarantees for their outputs. We propose a sequential conformal prediction method producing prediction sets that satisfy a rigorous statistical guarantee. This guarantee states that with high probability, the prediction sets contain at least one admissible (or valid) example.
  arXiv  Detail & Related papers  (2024-10-02T15:26:52Z)
• Does confidence calibration improve conformal prediction? [10.340903334800787]
  We show that current confidence calibration methods lead to larger prediction sets in adaptive conformal prediction. By investigating the role of temperature value, we observe that high-confidence predictions can enhance the efficiency of adaptive conformal prediction. We propose Conformal Temperature Scaling (ConfTS), a variant of temperature scaling with a novel loss function designed to enhance the efficiency of prediction sets.
  arXiv  Detail & Related papers  (2024-02-06T19:27:48Z)
• Likelihood Ratio Confidence Sets for Sequential Decision Making [51.66638486226482]
  We revisit the likelihood-based inference principle and propose to use likelihood ratios to construct valid confidence sequences. Our method is especially suitable for problems with well-specified likelihoods. We show how to provably choose the best sequence of estimators and shed light on connections to online convex optimization.
  arXiv  Detail & Related papers  (2023-11-08T00:10:21Z)
• Federated Conformal Predictors for Distributed Uncertainty Quantification [83.50609351513886]
  Conformal prediction is emerging as a popular paradigm for providing rigorous uncertainty quantification in machine learning. In this paper, we extend conformal prediction to the federated learning setting. We propose a weaker notion of partial exchangeability, better suited to the FL setting, and use it to develop the Federated Conformal Prediction framework.
  arXiv  Detail & Related papers  (2023-05-27T19:57:27Z)
• Debiased Fine-Tuning for Vision-language Models by Prompt Regularization [50.41984119504716]
  We present a new paradigm for fine-tuning large-scale vision pre-trained models on downstream task, dubbed Prompt Regularization (ProReg) ProReg uses the prediction by prompting the pretrained model to regularize the fine-tuning. We show the consistently strong performance of ProReg compared with conventional fine-tuning, zero-shot prompt, prompt tuning, and other state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-01-29T11:53:55Z)
• Few-Shot Calibration of Set Predictors via Meta-Learned Cross-Validation-Based Conformal Prediction [33.33774397643919]
  This paper introduces a novel meta-learning solution that aims at reducing the set prediction size. It builds on cross-validation-based CP, rather than the less efficient validation-based CP. It preserves formal per-task calibration guarantees, rather than less stringent task-marginal guarantees.
  arXiv  Detail & Related papers  (2022-10-06T17:21:03Z)
• Predictive Inference with Feature Conformal Prediction [80.77443423828315]
  We propose feature conformal prediction, which extends the scope of conformal prediction to semantic feature spaces. From a theoretical perspective, we demonstrate that feature conformal prediction provably outperforms regular conformal prediction under mild assumptions. Our approach could be combined with not only vanilla conformal prediction, but also other adaptive conformal prediction methods.
  arXiv  Detail & Related papers  (2022-10-01T02:57:37Z)
• Distribution-free binary classification: prediction sets, confidence intervals and calibration [106.50279469344937]
  We study three notions of uncertainty quantification -- calibration, confidence intervals and prediction sets -- for binary classification in the distribution-free setting. We derive confidence intervals for binned probabilities for both fixed-width and uniform-mass binning. As a consequence of our 'tripod' theorems, these confidence intervals for binned probabilities lead to distribution-free calibration.
  arXiv  Detail & Related papers  (2020-06-18T14:17:29Z)
• Tuning-free ridge estimators for high-dimensional generalized linear models [3.383670923637875]
  We show that ridge estimators can be modified such that tuning parameters can be avoided altogether. We also show that these modified versions can improve on the empirical prediction accuracies of standard ridge estimators combined with cross-validation.
  arXiv  Detail & Related papers  (2020-02-27T05:01:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.