Calibrated Multivariate Regression with Localized PIT Mappings

• URL: http://arxiv.org/abs/2409.10855v1
• Date: Tue, 17 Sep 2024 02:41:03 GMT
• Title: Calibrated Multivariate Regression with Localized PIT Mappings
• Authors: Lucas Kock, G. S. Rodrigues, Scott A. Sisson, Nadja Klein, David J. Nott,
• Abstract summary: This paper introduces a novel post-hoc recalibration approach that addresses multivariate calibration for potentially misspecified models. We present two versions of our approach: one uses K-nearest neighbors, and the other uses normalizing flows. We demonstrate the effectiveness of our approach on two real data applications: recalibrating a deep neural network's currency exchange rate forecast and improving a regression model for childhood malnutrition in India.
• Score: 4.277516034244117
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Calibration ensures that predicted uncertainties align with observed uncertainties. While there is an extensive literature on recalibration methods for univariate probabilistic forecasts, work on calibration for multivariate forecasts is much more limited. This paper introduces a novel post-hoc recalibration approach that addresses multivariate calibration for potentially misspecified models. Our method involves constructing local mappings between vectors of marginal probability integral transform values and the space of observations, providing a flexible and model free solution applicable to continuous, discrete, and mixed responses. We present two versions of our approach: one uses K-nearest neighbors, and the other uses normalizing flows. Each method has its own strengths in different situations. We demonstrate the effectiveness of our approach on two real data applications: recalibrating a deep neural network's currency exchange rate forecast and improving a regression model for childhood malnutrition in India for which the multivariate response has both discrete and continuous components.

Related papers

• Multivariate root-n-consistent smoothing parameter free matching estimators and estimators of inverse density weighted expectations [51.000851088730684]
  We develop novel modifications of nearest-neighbor and matching estimators which converge at the parametric $sqrt n $-rate. We stress that our estimators do not involve nonparametric function estimators and in particular do not rely on sample-size dependent parameters smoothing.
  arXiv  Detail & Related papers  (2024-07-11T13:28:34Z)
• Robust scalable initialization for Bayesian variational inference with multi-modal Laplace approximations [0.0]
  Variational mixtures with full-covariance structures suffer from a quadratic growth due to variational parameters with the number of parameters. We propose a method for constructing an initial Gaussian model approximation that can be used to warm-start variational inference.
  arXiv  Detail & Related papers  (2023-07-12T19:30:04Z)
• Generative machine learning methods for multivariate ensemble post-processing [2.266704492832475]
  We present a novel class of nonparametric data-driven distributional regression models based on generative machine learning. In two case studies, our generative model shows significant improvements over state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-09-26T09:02:30Z)
• Robust Calibration with Multi-domain Temperature Scaling [86.07299013396059]
  We develop a systematic calibration model to handle distribution shifts by leveraging data from multiple domains. Our proposed method -- multi-domain temperature scaling -- uses the robustness in the domains to improve calibration under distribution shift.
  arXiv  Detail & Related papers  (2022-06-06T17:32:12Z)
• Trustworthy Multimodal Regression with Mixture of Normal-inverse Gamma Distributions [91.63716984911278]
  We introduce a novel Mixture of Normal-Inverse Gamma distributions (MoNIG) algorithm, which efficiently estimates uncertainty in principle for adaptive integration of different modalities and produces a trustworthy regression result. Experimental results on both synthetic and different real-world data demonstrate the effectiveness and trustworthiness of our method on various multimodal regression tasks.
  arXiv  Detail & Related papers  (2021-11-11T14:28:12Z)
• Multivariate Probabilistic Regression with Natural Gradient Boosting [63.58097881421937]
  We propose a Natural Gradient Boosting (NGBoost) approach based on nonparametrically modeling the conditional parameters of the multivariate predictive distribution. Our method is robust, works out-of-the-box without extensive tuning, is modular with respect to the assumed target distribution, and performs competitively in comparison to existing approaches.
  arXiv  Detail & Related papers  (2021-06-07T17:44:49Z)
• Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift [100.52588638477862]
  We develop an approximate Bayesian inference scheme based on posterior regularisation. We demonstrate the utility of our method in the context of transferring prognostic models of prostate cancer across globally diverse populations.
  arXiv  Detail & Related papers  (2020-06-26T13:50:19Z)
• Calibration of Neural Networks using Splines [51.42640515410253]
  Measuring calibration error amounts to comparing two empirical distributions. We introduce a binning-free calibration measure inspired by the classical Kolmogorov-Smirnov (KS) statistical test. Our method consistently outperforms existing methods on KS error as well as other commonly used calibration measures.
  arXiv  Detail & Related papers  (2020-06-23T07:18:05Z)
• Variational Variance: Simple, Reliable, Calibrated Heteroscedastic Noise Variance Parameterization [3.553493344868413]
  We propose critiques to test predictive mean and variance calibration and the predictive distribution's ability to generate sensible data. We find that our solution, to treat heteroscedastic variance variationally, sufficiently regularizes variance to pass these PPCs.
  arXiv  Detail & Related papers  (2020-06-08T19:58:35Z)
• Quantile Regularization: Towards Implicit Calibration of Regression Models [30.872605139672086]
  We present a method for calibrating regression models based on a novel quantile regularizer defined as the cumulative KL divergence between two CDFs. We show that the proposed quantile regularizer significantly improves calibration for regression models trained using approaches, such as Dropout VI and Deep Ensembles.
  arXiv  Detail & Related papers  (2020-02-28T16:53:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.