Conformal Risk Control

• URL: http://arxiv.org/abs/2208.02814v3
• Date: Sat, 29 Apr 2023 21:20:48 GMT
• Title: Conformal Risk Control
• Authors: Anastasios N. Angelopoulos and Stephen Bates and Adam Fisch and Lihua Lei and Tal Schuster
• Abstract summary: We extend conformal prediction to control the expected value of any monotone loss function. We also introduce extensions of the idea to distribution shift, quantile risk control, multiple and adversarial risk control, and expectations of U-statistics.
• Score: 20.65019607005074
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We extend conformal prediction to control the expected value of any monotone loss function. The algorithm generalizes split conformal prediction together with its coverage guarantee. Like conformal prediction, the conformal risk control procedure is tight up to an $\mathcal{O}(1/n)$ factor. We also introduce extensions of the idea to distribution shift, quantile risk control, multiple and adversarial risk control, and expectations of U-statistics. Worked examples from computer vision and natural language processing demonstrate the usage of our algorithm to bound the false negative rate, graph distance, and token-level F1-score.

Related papers

• Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filtering [55.15192437680943]
  Generative models lack rigorous statistical guarantees for their outputs. We propose a sequential conformal prediction method producing prediction sets that satisfy a rigorous statistical guarantee. This guarantee states that with high probability, the prediction sets contain at least one admissible (or valid) example.
  arXiv  Detail & Related papers  (2024-10-02T15:26:52Z)
• Forking Uncertainties: Reliable Prediction and Model Predictive Control with Sequence Models via Conformal Risk Control [40.918012779935246]
  We introduce a novel post-hoc calibration procedure that operates on the predictions produced by any pre-designed probabilistic forecaster to yield reliable error bars. Unlike the state of the art, PTS-CRC can satisfy reliability definitions beyond coverage. We experimentally validate the performance of PTS-CRC prediction and control by studying a number of use cases in the context of wireless networking.
  arXiv  Detail & Related papers  (2023-10-16T11:35:41Z)
• Conformal PID Control for Time Series Prediction [10.992151305603265]
  We study the problem of uncertainty quantification for time series prediction. We present algorithms that prospectively model conformal scores in an online setting. We also run experiments on predicting electricity demand, market returns, and temperature.
  arXiv  Detail & Related papers  (2023-07-31T17:59:16Z)
• On Regularization and Inference with Label Constraints [62.60903248392479]
  We compare two strategies for encoding label constraints in a machine learning pipeline, regularization with constraints and constrained inference. For regularization, we show that it narrows the generalization gap by precluding models that are inconsistent with the constraints. For constrained inference, we show that it reduces the population risk by correcting a model's violation, and hence turns the violation into an advantage.
  arXiv  Detail & Related papers  (2023-07-08T03:39:22Z)
• Loss-Controlling Calibration for Predictive Models [5.51361762392299]
  We propose a learning framework for calibrating predictive models to make loss-controlling prediction for exchangeable data. By comparison, the predictors built by the proposed loss-controlling approach are not limited to set predictors. Our proposed method is applied to selective regression and high-impact weather forecasting problems.
  arXiv  Detail & Related papers  (2023-01-11T09:44:55Z)
• Conformal Loss-Controlling Prediction [23.218535051437588]
  Conformal prediction is a learning framework controlling prediction coverage of prediction sets. This work proposes a learning framework named conformal loss-controlling prediction, which extends conformal prediction to the situation where the value of a loss function needs to be controlled.
  arXiv  Detail & Related papers  (2023-01-06T08:58:49Z)
• Tailoring to the Tails: Risk Measures for Fine-Grained Tail Sensitivity [10.482805367361818]
  Expected risk rearrangement (ERM) is at the core of machine learning systems. We propose a general approach to construct risk measures which exhibit a desired tail sensitivity.
  arXiv  Detail & Related papers  (2022-08-05T09:51:18Z)
• Mitigating multiple descents: A model-agnostic framework for risk monotonization [84.6382406922369]
  We develop a general framework for risk monotonization based on cross-validation. We propose two data-driven methodologies, namely zero- and one-step, that are akin to bagging and boosting.
  arXiv  Detail & Related papers  (2022-05-25T17:41:40Z)
• Efficient and Differentiable Conformal Prediction with General Function Classes [96.74055810115456]
  We propose a generalization of conformal prediction to multiple learnable parameters. We show that it achieves approximate valid population coverage and near-optimal efficiency within class. Experiments show that our algorithm is able to learn valid prediction sets and improve the efficiency significantly.
  arXiv  Detail & Related papers  (2022-02-22T18:37:23Z)
• Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control [67.52000805944924]
  Learn then Test (LTT) is a framework for calibrating machine learning models. Our main insight is to reframe the risk-control problem as multiple hypothesis testing. We use our framework to provide new calibration methods for several core machine learning tasks with detailed worked examples in computer vision.
  arXiv  Detail & Related papers  (2021-10-03T17:42:03Z)
• Certification of Iterative Predictions in Bayesian Neural Networks [79.15007746660211]
  We compute lower bounds for the probability that trajectories of the BNN model reach a given set of states while avoiding a set of unsafe states. We use the lower bounds in the context of control and reinforcement learning to provide safety certification for given control policies.
  arXiv  Detail & Related papers  (2021-05-21T05:23:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.