HopCast: Calibration of Autoregressive Dynamics Models

• URL: http://arxiv.org/abs/2501.16587v3
• Date: Fri, 23 May 2025 16:45:54 GMT
• Title: HopCast: Calibration of Autoregressive Dynamics Models
• Authors: Muhammad Bilal Shahid, Cody Fleming,
• Abstract summary: This work introduces an alternative Predictor-Corrector approach named hop that uses Modern Hopfield Networks (MHN) to learn the errors of a deterministic Predictor.<n>The Corrector predicts a set of errors for the Predictor's output based on a context state at any timestep during autoregression.<n>The calibration and prediction performances are evaluated across a set of dynamical systems.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep learning models are often trained to approximate dynamical systems that can be modeled using differential equations. Many of these models are optimized to predict one step ahead; such approaches produce calibrated one-step predictions if the predictive model can quantify uncertainty, such as Deep Ensembles. At inference time, multi-step predictions are generated via autoregression, which needs a sound uncertainty propagation method to produce calibrated multi-step predictions. This work introduces an alternative Predictor-Corrector approach named \hop{} that uses Modern Hopfield Networks (MHN) to learn the errors of a deterministic Predictor that approximates the dynamical system. The Corrector predicts a set of errors for the Predictor's output based on a context state at any timestep during autoregression. The set of errors creates sharper and well-calibrated prediction intervals with higher predictive accuracy compared to baselines without uncertainty propagation. The calibration and prediction performances are evaluated across a set of dynamical systems. This work is also the first to benchmark existing uncertainty propagation methods based on calibration errors.

Related papers

• Awareness of uncertainty in classification using a multivariate model and multi-views [1.3048920509133808]
  The proposed model regularizes uncertain predictions, and trains to calculate both the predictions and their uncertainty estimations. Given the multi-view predictions together with their uncertainties and confidences, we proposed several methods to calculate final predictions. The proposed methodology was tested using CIFAR-10 dataset with clean and noisy labels.
  arXiv  Detail & Related papers  (2024-04-16T06:40:51Z)
• Conformal online model aggregation [29.43493007296859]
  This paper proposes a new approach towards conformal model aggregation in online settings. It is based on combining the prediction sets from several algorithms by voting, where weights on the models are adapted over time based on past performance.
  arXiv  Detail & Related papers  (2024-03-22T15:40:06Z)
• Faster Recalibration of an Online Predictor via Approachability [12.234317585724868]
  We introduce a technique for taking an online predictive model which might not be calibrated and transforming its predictions to calibrated predictions without much increase to the loss of the original model. Our proposed algorithm achieves calibration and accuracy at a faster rate than existing techniques arXiv:1607.03594 and is the first algorithm to offer a flexible tradeoff between calibration error and accuracy in the online setting.
  arXiv  Detail & Related papers  (2023-10-25T20:59:48Z)
• Multiclass Alignment of Confidence and Certainty for Network Calibration [10.15706847741555]
  Recent studies reveal that deep neural networks (DNNs) are prone to making overconfident predictions. We propose a new train-time calibration method, which features a simple, plug-and-play auxiliary loss known as multi-class alignment of predictive mean confidence and predictive certainty (MACC) Our method achieves state-of-the-art calibration performance for both in-domain and out-domain predictions.
  arXiv  Detail & Related papers  (2023-09-06T00:56:24Z)
• Better Batch for Deep Probabilistic Time Series Forecasting [15.31488551912888]
  We propose an innovative training method that incorporates error autocorrelation to enhance probabilistic forecasting accuracy. Our method constructs a mini-batch as a collection of $D$ consecutive time series segments for model training. It explicitly learns a time-varying covariance matrix over each mini-batch, encoding error correlation among adjacent time steps.
  arXiv  Detail & Related papers  (2023-05-26T15:36:59Z)
• Learning Sample Difficulty from Pre-trained Models for Reliable Prediction [55.77136037458667]
  We propose to utilize large-scale pre-trained models to guide downstream model training with sample difficulty-aware entropy regularization. We simultaneously improve accuracy and uncertainty calibration across challenging benchmarks.
  arXiv  Detail & Related papers  (2023-04-20T07:29:23Z)
• Sharp Calibrated Gaussian Processes [58.94710279601622]
  State-of-the-art approaches for designing calibrated models rely on inflating the Gaussian process posterior variance. We present a calibration approach that generates predictive quantiles using a computation inspired by the vanilla Gaussian process posterior variance. Our approach is shown to yield a calibrated model under reasonable assumptions.
  arXiv  Detail & Related papers  (2023-02-23T12:17:36Z)
• Stabilizing Machine Learning Prediction of Dynamics: Noise and Noise-inspired Regularization [58.720142291102135]
  Recent has shown that machine learning (ML) models can be trained to accurately forecast the dynamics of chaotic dynamical systems. In the absence of mitigating techniques, this technique can result in artificially rapid error growth, leading to inaccurate predictions and/or climate instability. We introduce Linearized Multi-Noise Training (LMNT), a regularization technique that deterministically approximates the effect of many small, independent noise realizations added to the model input during training.
  arXiv  Detail & Related papers  (2022-11-09T23:40:52Z)
• Correcting Model Bias with Sparse Implicit Processes [0.9187159782788579]
  We show that Sparse Implicit Processes (SIP) is capable of correcting model bias when the data generating mechanism differs strongly from the one implied by the model. We use synthetic datasets to show that SIP is capable of providing predictive distributions that reflect the data better than the exact predictions of the initial, but wrongly assumed model.
  arXiv  Detail & Related papers  (2022-07-21T18:00:01Z)
• T-Cal: An optimal test for the calibration of predictive models [49.11538724574202]
  We consider detecting mis-calibration of predictive models using a finite validation dataset as a hypothesis testing problem. detecting mis-calibration is only possible when the conditional probabilities of the classes are sufficiently smooth functions of the predictions. We propose T-Cal, a minimax test for calibration based on a de-biased plug-in estimator of the $ell$-Expected Error (ECE)
  arXiv  Detail & Related papers  (2022-03-03T16:58:54Z)
• Autoregressive Quantile Flows for Predictive Uncertainty Estimation [7.184701179854522]
  We propose Autoregressive Quantile Flows, a flexible class of probabilistic models over high-dimensional variables. These models are instances of autoregressive flows trained using a novel objective based on proper scoring rules.
  arXiv  Detail & Related papers  (2021-12-09T01:11:26Z)
• Dense Uncertainty Estimation [62.23555922631451]
  In this paper, we investigate neural networks and uncertainty estimation techniques to achieve both accurate deterministic prediction and reliable uncertainty estimation. We work on two types of uncertainty estimations solutions, namely ensemble based methods and generative model based methods, and explain their pros and cons while using them in fully/semi/weakly-supervised framework.
  arXiv  Detail & Related papers  (2021-10-13T01:23:48Z)
• Quantifying Model Predictive Uncertainty with Perturbation Theory [21.591460685054546]
  We propose a framework for predictive uncertainty quantification of a neural network. We use perturbation theory from quantum physics to formulate a moment decomposition problem. Our approach provides fast model predictive uncertainty estimates with much greater precision and calibration.
  arXiv  Detail & Related papers  (2021-09-22T17:55:09Z)
• Anomaly Detection of Time Series with Smoothness-Inducing Sequential Variational Auto-Encoder [59.69303945834122]
  We present a Smoothness-Inducing Sequential Variational Auto-Encoder (SISVAE) model for robust estimation and anomaly detection of time series. Our model parameterizes mean and variance for each time-stamp with flexible neural networks. We show the effectiveness of our model on both synthetic datasets and public real-world benchmarks.
  arXiv  Detail & Related papers  (2021-02-02T06:15:15Z)
• Learning Interpretable Deep State Space Model for Probabilistic Time Series Forecasting [98.57851612518758]
  Probabilistic time series forecasting involves estimating the distribution of future based on its history. We propose a deep state space model for probabilistic time series forecasting whereby the non-linear emission model and transition model are parameterized by networks. We show in experiments that our model produces accurate and sharp probabilistic forecasts.
  arXiv  Detail & Related papers  (2021-01-31T06:49:33Z)
• Generative Temporal Difference Learning for Infinite-Horizon Prediction [101.59882753763888]
  We introduce the $gamma$-model, a predictive model of environment dynamics with an infinite probabilistic horizon. We discuss how its training reflects an inescapable tradeoff between training-time and testing-time compounding errors.
  arXiv  Detail & Related papers  (2020-10-27T17:54:12Z)
• AutoCP: Automated Pipelines for Accurate Prediction Intervals [84.16181066107984]
  This paper proposes an AutoML framework called Automatic Machine Learning for Conformal Prediction (AutoCP) Unlike the familiar AutoML frameworks that attempt to select the best prediction model, AutoCP constructs prediction intervals that achieve the user-specified target coverage rate. We tested AutoCP on a variety of datasets and found that it significantly outperforms benchmark algorithms.
  arXiv  Detail & Related papers  (2020-06-24T23:13:11Z)
• Individual Calibration with Randomized Forecasting [116.2086707626651]
  We show that calibration for individual samples is possible in the regression setup if the predictions are randomized. We design a training objective to enforce individual calibration and use it to train randomized regression functions.
  arXiv  Detail & Related papers  (2020-06-18T05:53:10Z)
• Quantile Regularization: Towards Implicit Calibration of Regression Models [30.872605139672086]
  We present a method for calibrating regression models based on a novel quantile regularizer defined as the cumulative KL divergence between two CDFs. We show that the proposed quantile regularizer significantly improves calibration for regression models trained using approaches, such as Dropout VI and Deep Ensembles.
  arXiv  Detail & Related papers  (2020-02-28T16:53:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.