Conformal Methods for Quantifying Uncertainty in Spatiotemporal Data: A
Survey
- URL: http://arxiv.org/abs/2209.03580v1
- Date: Thu, 8 Sep 2022 06:08:48 GMT
- Title: Conformal Methods for Quantifying Uncertainty in Spatiotemporal Data: A
Survey
- Authors: Sophia Sun
- Abstract summary: In high-risk settings, it is important that a model produces uncertainty to reflect its own confidence and avoid failures.
In this paper we survey recent works on uncertainty (UQ) for deep learning, in particular distribution-free Conformal Prediction method for its mathematical and wide applicability.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Machine learning methods are increasingly widely used in high-risk settings
such as healthcare, transportation, and finance. In these settings, it is
important that a model produces calibrated uncertainty to reflect its own
confidence and avoid failures. In this paper we survey recent works on
uncertainty quantification (UQ) for deep learning, in particular
distribution-free Conformal Prediction method for its mathematical properties
and wide applicability. We will cover the theoretical guarantees of conformal
methods, introduce techniques that improve calibration and efficiency for UQ in
the context of spatiotemporal data, and discuss the role of UQ in the context
of safe decision making.
Related papers
- Conditionally valid Probabilistic Conformal Prediction [57.80927226809277]
We develop a new method for creating prediction sets that combines the flexibility of conformal methods with an estimate of the conditional distribution.
We demonstrate the effectiveness of our approach through extensive simulations, showing that it outperforms existing methods in terms of conditional coverage.
arXiv Detail & Related papers (2024-07-01T20:44:48Z) - ConU: Conformal Uncertainty in Large Language Models with Correctness Coverage Guarantees [68.33498595506941]
Uncertainty quantification in natural language generation (NLG) tasks remains an open challenge.
This study investigates adapting conformal prediction (CP), which can convert any measure of uncertainty into rigorous theoretical guarantees.
We propose a sampling-based uncertainty measure leveraging self-consistency and develop a conformal uncertainty criterion.
arXiv Detail & Related papers (2024-06-29T17:33:07Z) - Uncertainty Quantification for Forward and Inverse Problems of PDEs via
Latent Global Evolution [110.99891169486366]
We propose a method that integrates efficient and precise uncertainty quantification into a deep learning-based surrogate model.
Our method endows deep learning-based surrogate models with robust and efficient uncertainty quantification capabilities for both forward and inverse problems.
Our method excels at propagating uncertainty over extended auto-regressive rollouts, making it suitable for scenarios involving long-term predictions.
arXiv Detail & Related papers (2024-02-13T11:22:59Z) - Calibration in Machine Learning Uncertainty Quantification: beyond
consistency to target adaptivity [0.0]
This article aims to show that consistency and adaptivity are complementary validation targets, and that a good consistency does not imply a good adaptivity.
Adapted validation methods are proposed and illustrated on a representative example.
arXiv Detail & Related papers (2023-09-12T13:58:04Z) - Conformal Prediction for Federated Uncertainty Quantification Under
Label Shift [57.54977668978613]
Federated Learning (FL) is a machine learning framework where many clients collaboratively train models.
We develop a new conformal prediction method based on quantile regression and take into account privacy constraints.
arXiv Detail & Related papers (2023-06-08T11:54:58Z) - Validation of uncertainty quantification metrics: a primer based on the
consistency and adaptivity concepts [0.0]
The study is conceived as an introduction to UQ validation, and all methods are derived from a few basic rules.
The methods are illustrated and tested on synthetic datasets and representative examples extracted from the recent physico-chemical machine learning UQ literature.
arXiv Detail & Related papers (2023-03-13T15:13:03Z) - Uncertainty Estimation based on Geometric Separation [13.588210692213568]
In machine learning, accurately predicting the probability that a specific input is correct is crucial for risk management.
We put forward a novel geometric-based approach for improving uncertainty estimations in machine learning models.
arXiv Detail & Related papers (2023-01-11T13:19:24Z) - Quantifying Uncertainty in Deep Spatiotemporal Forecasting [67.77102283276409]
We describe two types of forecasting problems: regular grid-based and graph-based.
We analyze UQ methods from both the Bayesian and the frequentist point view, casting in a unified framework via statistical decision theory.
Through extensive experiments on real-world road network traffic, epidemics, and air quality forecasting tasks, we reveal the statistical computational trade-offs for different UQ methods.
arXiv Detail & Related papers (2021-05-25T14:35:46Z) - Probabilistic robust linear quadratic regulators with Gaussian processes [73.0364959221845]
Probabilistic models such as Gaussian processes (GPs) are powerful tools to learn unknown dynamical systems from data for subsequent use in control design.
We present a novel controller synthesis for linearized GP dynamics that yields robust controllers with respect to a probabilistic stability margin.
arXiv Detail & Related papers (2021-05-17T08:36:18Z) - Empirical Frequentist Coverage of Deep Learning Uncertainty
Quantification Procedures [13.890139530120164]
We provide the first large scale evaluation of the empirical frequentist coverage properties of uncertainty quantification techniques.
We find that, in general, some methods do achieve desirable coverage properties on in distribution samples, but that coverage is not maintained on out-of-distribution data.
arXiv Detail & Related papers (2020-10-06T21:22:46Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.