Quantile Risk Control: A Flexible Framework for Bounding the Probability of High-Loss Predictions

• URL: http://arxiv.org/abs/2212.13629v1
• Date: Tue, 27 Dec 2022 22:08:29 GMT
• Title: Quantile Risk Control: A Flexible Framework for Bounding the Probability of High-Loss Predictions
• Authors: Jake C. Snell, Thomas P. Zollo, Zhun Deng, Toniann Pitassi and Richard Zemel
• Abstract summary: We propose a flexible framework to produce a family of bounds on quantiles of the loss distribution incurred by a predictor. We show that a quantile is an informative way of quantifying predictive performance, and that our framework applies to a variety of quantile-based metrics.
• Score: 11.842061466957686
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Rigorous guarantees about the performance of predictive algorithms are necessary in order to ensure their responsible use. Previous work has largely focused on bounding the expected loss of a predictor, but this is not sufficient in many risk-sensitive applications where the distribution of errors is important. In this work, we propose a flexible framework to produce a family of bounds on quantiles of the loss distribution incurred by a predictor. Our method takes advantage of the order statistics of the observed loss values rather than relying on the sample mean alone. We show that a quantile is an informative way of quantifying predictive performance, and that our framework applies to a variety of quantile-based metrics, each targeting important subsets of the data distribution. We analyze the theoretical properties of our proposed method and demonstrate its ability to rigorously control loss quantiles on several real-world datasets.

Related papers

• Semiparametric conformal prediction [79.6147286161434]
  Risk-sensitive applications require well-calibrated prediction sets over multiple, potentially correlated target variables. We treat the scores as random vectors and aim to construct the prediction set accounting for their joint correlation structure. We report desired coverage and competitive efficiency on a range of real-world regression problems.
  arXiv  Detail & Related papers  (2024-11-04T14:29:02Z)
• Relaxed Quantile Regression: Prediction Intervals for Asymmetric Noise [51.87307904567702]
  Quantile regression is a leading approach for obtaining such intervals via the empirical estimation of quantiles in the distribution of outputs. We propose Relaxed Quantile Regression (RQR), a direct alternative to quantile regression based interval construction that removes this arbitrary constraint. We demonstrate that this added flexibility results in intervals with an improvement in desirable qualities.
  arXiv  Detail & Related papers  (2024-06-05T13:36:38Z)
• Distribution-Free Statistical Dispersion Control for Societal Applications [16.43522470711466]
  Explicit finite-sample statistical guarantees on model performance are an important ingredient in responsible machine learning. Previous work has focused mainly on bounding either the expected loss of a predictor or the probability that an individual prediction will incur a loss value in a specified range. We propose a simple yet flexible framework that allows us to handle a much richer class of statistical functionals beyond previous work.
  arXiv  Detail & Related papers  (2023-09-25T00:31:55Z)
• Quantification of Predictive Uncertainty via Inference-Time Sampling [57.749601811982096]
  We propose a post-hoc sampling strategy for estimating predictive uncertainty accounting for data ambiguity. The method can generate different plausible outputs for a given input and does not assume parametric forms of predictive distributions.
  arXiv  Detail & Related papers  (2023-08-03T12:43:21Z)
• Conformal Prediction with Missing Values [19.18178194789968]
  We first show that the marginal coverage guarantee of conformal prediction holds on imputed data for any missingness distribution. We then show that a universally consistent quantile regression algorithm trained on the imputed data is Bayes optimal for the pinball risk.
  arXiv  Detail & Related papers  (2023-06-05T09:28:03Z)
• Toward Robust Uncertainty Estimation with Random Activation Functions [3.0586855806896045]
  We propose a novel approach for uncertainty quantification via ensembles, called Random Activation Functions (RAFs) Ensemble. RAFs Ensemble outperforms state-of-the-art ensemble uncertainty quantification methods on both synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2023-02-28T13:17:56Z)
• Conformal Loss-Controlling Prediction [23.218535051437588]
  Conformal prediction is a learning framework controlling prediction coverage of prediction sets. This work proposes a learning framework named conformal loss-controlling prediction, which extends conformal prediction to the situation where the value of a loss function needs to be controlled.
  arXiv  Detail & Related papers  (2023-01-06T08:58:49Z)
• Interpretable Social Anchors for Human Trajectory Forecasting in Crowds [84.20437268671733]
  We propose a neural network-based system to predict human trajectory in crowds. We learn interpretable rule-based intents, and then utilise the expressibility of neural networks to model scene-specific residual. Our architecture is tested on the interaction-centric benchmark TrajNet++.
  arXiv  Detail & Related papers  (2021-05-07T09:22:34Z)
• Estimation and Applications of Quantiles in Deep Binary Classification [0.0]
  Quantile regression, based on check loss, is a widely used inferential paradigm in Statistics. We consider the analogue of check loss in the binary classification setting. We develop individualized confidence scores that can be used to decide whether a prediction is reliable.
  arXiv  Detail & Related papers  (2021-02-09T07:07:42Z)
• Distribution-Free, Risk-Controlling Prediction Sets [112.9186453405701]
  We show how to generate set-valued predictions from a black-box predictor that control the expected loss on future test points at a user-specified level. Our approach provides explicit finite-sample guarantees for any dataset by using a holdout set to calibrate the size of the prediction sets.
  arXiv  Detail & Related papers  (2021-01-07T18:59:33Z)
• GenDICE: Generalized Offline Estimation of Stationary Values [108.17309783125398]
  We show that effective estimation can still be achieved in important applications. Our approach is based on estimating a ratio that corrects for the discrepancy between the stationary and empirical distributions. The resulting algorithm, GenDICE, is straightforward and effective.
  arXiv  Detail & Related papers  (2020-02-21T00:27:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.