Recalibrating probabilistic forecasts of epidemics
- URL: http://arxiv.org/abs/2112.06305v1
- Date: Sun, 12 Dec 2021 19:22:24 GMT
- Title: Recalibrating probabilistic forecasts of epidemics
- Authors: Aaron Rumack, Ryan J. Tibshirani, Roni Rosenfeld
- Abstract summary: We present a recalibration method that can be applied to a black-box forecaster given retrospective forecasts and observations.
This method is guaranteed to improve calibration and log score performance when trained and measured in-sample.
We apply this recalibration method to the 27 influenza forecasters in the FluSight Network and show that recalibration reliably improves forecast accuracy and calibration.
- Score: 13.447680826767183
- License: http://creativecommons.org/licenses/by-sa/4.0/
- Abstract: Distributional forecasts are important for a wide variety of applications,
including forecasting epidemics. Often, forecasts are miscalibrated, or
unreliable in assigning uncertainty to future events. We present a
recalibration method that can be applied to a black-box forecaster given
retrospective forecasts and observations, as well as an extension to make this
method more effective in recalibrating epidemic forecasts. This method is
guaranteed to improve calibration and log score performance when trained and
measured in-sample. We also prove that the increase in expected log score of a
recalibrated forecaster is equal to the entropy of the PIT distribution. We
apply this recalibration method to the 27 influenza forecasters in the FluSight
Network and show that recalibration reliably improves forecast accuracy and
calibration. This method is effective, robust, and easy to use as a
post-processing tool to improve epidemic forecasts.
Related papers
- Modeling and Predicting Epidemic Spread: A Gaussian Process Regression
Approach [0.805635934199494]
We present a new method based on Gaussian Process Regression to model and predict epidemics.
We develop a novel bound on the variance of the prediction that quantifies the impact of the epidemic data.
We quantify how the epidemic spread, the infection data, and the length of the prediction horizon all affect this error bound.
arXiv Detail & Related papers (2023-12-14T22:45:01Z) - When Rigidity Hurts: Soft Consistency Regularization for Probabilistic
Hierarchical Time Series Forecasting [69.30930115236228]
Probabilistic hierarchical time-series forecasting is an important variant of time-series forecasting.
Most methods focus on point predictions and do not provide well-calibrated probabilistic forecasts distributions.
We propose PROFHiT, a fully probabilistic hierarchical forecasting model that jointly models forecast distribution of entire hierarchy.
arXiv Detail & Related papers (2023-10-17T20:30:16Z) - Performative Time-Series Forecasting [71.18553214204978]
We formalize performative time-series forecasting (PeTS) from a machine-learning perspective.
We propose a novel approach, Feature Performative-Shifting (FPS), which leverages the concept of delayed response to anticipate distribution shifts.
We conduct comprehensive experiments using multiple time-series models on COVID-19 and traffic forecasting tasks.
arXiv Detail & Related papers (2023-10-09T18:34:29Z) - Parity Calibration [5.768816587293478]
We introduce the notion of parity calibration, which captures the goal of forecasting for the increase-decrease event in a timeseries.
We show that such a strategy leads to theoretical unpredictability and poor practical performance.
We demonstrate the effectiveness of our approach on real-world case studies in epidemiology, weather forecasting, and model-based control in nuclear fusion.
arXiv Detail & Related papers (2023-05-29T23:27:42Z) - Forecast reconciliation for vaccine supply chain optimization [61.13962963550403]
Vaccine supply chain optimization can benefit from hierarchical time series forecasting.
Forecasts of different hierarchy levels become incoherent when higher levels do not match the sum of the lower levels forecasts.
We tackle the vaccine sale forecasting problem by modeling sales data from GSK between 2010 and 2021 as a hierarchical time series.
arXiv Detail & Related papers (2023-05-02T14:34:34Z) - Forecast Hedging and Calibration [8.858351266850544]
We develop the concept of forecast hedging, which consists of choosing the forecasts so as to guarantee the expected track record can only improve.
This yields all the calibration results by the same simple argument while differentiating between them by the forecast-hedging tools used.
Additional contributions are an improved definition of continuous calibration, ensuing game dynamics that yield Nashlibria in the long run, and a new forecasting procedure for binary events that is simpler than all known such procedures.
arXiv Detail & Related papers (2022-10-13T16:48:25Z) - When Rigidity Hurts: Soft Consistency Regularization for Probabilistic
Hierarchical Time Series Forecasting [69.30930115236228]
Probabilistic hierarchical time-series forecasting is an important variant of time-series forecasting.
Most methods focus on point predictions and do not provide well-calibrated probabilistic forecasts distributions.
We propose PROFHiT, a fully probabilistic hierarchical forecasting model that jointly models forecast distribution of entire hierarchy.
arXiv Detail & Related papers (2022-06-16T06:13:53Z) - Forecast Evaluation in Large Cross-Sections of Realized Volatility [0.0]
We evaluate the predictive accuracy of the model based on the augmented cross-section when forecasting Realized volatility.
We study the sensitivity of forecasts to the model specification by incorporating a measurement error correction as well as cross-sectional jump component measures.
arXiv Detail & Related papers (2021-12-09T13:19:09Z) - When in Doubt: Neural Non-Parametric Uncertainty Quantification for
Epidemic Forecasting [70.54920804222031]
Most existing forecasting models disregard uncertainty quantification, resulting in mis-calibrated predictions.
Recent works in deep neural models for uncertainty-aware time-series forecasting also have several limitations.
We model the forecasting task as a probabilistic generative process and propose a functional neural process model called EPIFNP.
arXiv Detail & Related papers (2021-06-07T18:31:47Z) - Improving seasonal forecast using probabilistic deep learning [1.1988695717766686]
We develop a probabilistic deep neural network model to enhance seasonal forecast capability and forecast diagnosis.
By leveraging complex physical relationships encoded in climate simulations, our model demonstrates favorable deterministic and probabilistic skill.
We give a more definitive answer to how the El Nino/Southern Oscillation, the dominant mode of seasonal variability, modulates global seasonal predictability.
arXiv Detail & Related papers (2020-10-27T21:02:26Z) - Learning to Predict Error for MRI Reconstruction [67.76632988696943]
We demonstrate that predictive uncertainty estimated by the current methods does not highly correlate with prediction error.
We propose a novel method that estimates the target labels and magnitude of the prediction error in two steps.
arXiv Detail & Related papers (2020-02-13T15:55:32Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.