Normalizing Flows for Conformal Regression

• URL: http://arxiv.org/abs/2406.03346v2
• Date: Wed, 26 Jun 2024 15:55:02 GMT
• Title: Normalizing Flows for Conformal Regression
• Authors: Nicolo Colombo,
• Abstract summary: Conformal Prediction (CP) algorithms estimate the uncertainty of a prediction model by calibrating its outputs on labeled data. We present a general scheme to localize the intervals by training the calibration process. Unlike the Error Reweighting CP algorithm of Papadopoulos et al. (2008), the framework allows estimating the gap between nominal and empirical conditional validity.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Conformal Prediction (CP) algorithms estimate the uncertainty of a prediction model by calibrating its outputs on labeled data. The same calibration scheme usually applies to any model and data without modifications. The obtained prediction intervals are valid by construction but could be inefficient, i.e. unnecessarily big, if the prediction errors are not uniformly distributed over the input space. We present a general scheme to localize the intervals by training the calibration process. The standard prediction error is replaced by an optimized distance metric that depends explicitly on the object attributes. Learning the optimal metric is equivalent to training a Normalizing Flow that acts on the joint distribution of the errors and the inputs. Unlike the Error Reweighting CP algorithm of Papadopoulos et al. (2008), the framework allows estimating the gap between nominal and empirical conditional validity. The approach is compatible with existing locally-adaptive CP strategies based on re-weighting the calibration samples and applies to any point-prediction model without retraining.

Related papers

• Beyond Calibration: Assessing the Probabilistic Fit of Neural Regressors via Conditional Congruence [2.2359781747539396]
  Deep networks often suffer from overconfidence and misaligned predictive distributions. We introduce a metric, Conditional Congruence Error (CCE), that uses conditional kernel mean embeddings to estimate the distance between the learned predictive distribution and the empirical, conditional distribution in a dataset. We show that using to measure congruence 1) accurately quantifies misalignment between distributions when the data generating process is known, 2) effectively scales to real-world, high dimensional image regression tasks, and 3) can be used to gauge model reliability on unseen instances.
  arXiv  Detail & Related papers  (2024-05-20T23:30:07Z)
• Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
  We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores. We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
  arXiv  Detail & Related papers  (2023-02-23T18:57:14Z)
• Few-Shot Calibration of Set Predictors via Meta-Learned Cross-Validation-Based Conformal Prediction [33.33774397643919]
  This paper introduces a novel meta-learning solution that aims at reducing the set prediction size. It builds on cross-validation-based CP, rather than the less efficient validation-based CP. It preserves formal per-task calibration guarantees, rather than less stringent task-marginal guarantees.
  arXiv  Detail & Related papers  (2022-10-06T17:21:03Z)
• Parametric and Multivariate Uncertainty Calibration for Regression and Object Detection [4.630093015127541]
  We show that common detection models overestimate the spatial uncertainty in comparison to the observed error. Our experiments show that the simple Isotonic Regression recalibration method is sufficient to achieve a good calibrated uncertainty. In contrast, if normal distributions are required for subsequent processes, our GP-Normal recalibration method yields the best results.
  arXiv  Detail & Related papers  (2022-07-04T08:00:20Z)
• Modular Conformal Calibration [80.33410096908872]
  We introduce a versatile class of algorithms for recalibration in regression. This framework allows one to transform any regression model into a calibrated probabilistic model. We conduct an empirical study of MCC on 17 regression datasets.
  arXiv  Detail & Related papers  (2022-06-23T03:25:23Z)
• CovarianceNet: Conditional Generative Model for Correct Covariance Prediction in Human Motion Prediction [71.31516599226606]
  We present a new method to correctly predict the uncertainty associated with the predicted distribution of future trajectories. Our approach, CovariaceNet, is based on a Conditional Generative Model with Gaussian latent variables.
  arXiv  Detail & Related papers  (2021-09-07T09:38:24Z)
• Cross-validation: what does it estimate and how well does it do it? [2.049702429898688]
  Cross-validation is a widely-used technique to estimate prediction error, but its behavior is complex and not fully understood. We prove that this is not the case for the linear model fit by ordinary least squares; rather it estimates the average prediction error of models fit on other unseen training sets drawn from the same population.
  arXiv  Detail & Related papers  (2021-04-01T17:58:54Z)
• Localized Calibration: Metrics and Recalibration [133.07044916594361]
  We propose a fine-grained calibration metric that spans the gap between fully global and fully individualized calibration. We then introduce a localized recalibration method, LoRe, that improves the LCE better than existing recalibration methods.
  arXiv  Detail & Related papers  (2021-02-22T07:22:12Z)
• AutoCP: Automated Pipelines for Accurate Prediction Intervals [84.16181066107984]
  This paper proposes an AutoML framework called Automatic Machine Learning for Conformal Prediction (AutoCP) Unlike the familiar AutoML frameworks that attempt to select the best prediction model, AutoCP constructs prediction intervals that achieve the user-specified target coverage rate. We tested AutoCP on a variety of datasets and found that it significantly outperforms benchmark algorithms.
  arXiv  Detail & Related papers  (2020-06-24T23:13:11Z)
• Individual Calibration with Randomized Forecasting [116.2086707626651]
  We show that calibration for individual samples is possible in the regression setup if the predictions are randomized. We design a training objective to enforce individual calibration and use it to train randomized regression functions.
  arXiv  Detail & Related papers  (2020-06-18T05:53:10Z)
• CRUDE: Calibrating Regression Uncertainty Distributions Empirically [4.552831400384914]
  Calibrated uncertainty estimates in machine learning are crucial to many fields such as autonomous vehicles, medicine, and weather and climate forecasting. We present a calibration method for regression settings that does not assume a particular uncertainty distribution over the error: Calibrating Regression Uncertainty Distributions Empirically (CRUDE) CRUDE demonstrates consistently sharper, better calibrated, and more accurate uncertainty estimates than state-of-the-art techniques.
  arXiv  Detail & Related papers  (2020-05-26T03:08:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.