Excess risk analysis for epistemic uncertainty with application to variational inference

• URL: http://arxiv.org/abs/2206.01606v1
• Date: Thu, 2 Jun 2022 12:12:24 GMT
• Title: Excess risk analysis for epistemic uncertainty with application to variational inference
• Authors: Futoshi Futami, Tomoharu Iwata, Naonori Ueda, Issei Sato, Masashi Sugiyama
• Abstract summary: We present a novel EU analysis in the frequentist setting, where data is generated from an unknown distribution. We show a relation between the generalization ability and the widely used EU measurements, such as the variance and entropy of the predictive distribution. We propose new variational inference that directly controls the prediction and EU evaluation performances based on the PAC-Bayesian theory.
• Score: 110.4676591819618
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We analyze the epistemic uncertainty (EU) of supervised learning in Bayesian inference by focusing on the excess risk. Existing analysis is limited to the Bayesian setting, which assumes a correct model and exact Bayesian posterior distribution. Thus we cannot apply the existing theory to modern Bayesian algorithms, such as variational inference. To address this, we present a novel EU analysis in the frequentist setting, where data is generated from an unknown distribution. We show a relation between the generalization ability and the widely used EU measurements, such as the variance and entropy of the predictive distribution. Then we show their convergence behaviors theoretically. Finally, we propose new variational inference that directly controls the prediction and EU evaluation performances based on the PAC-Bayesian theory. Numerical experiments show that our algorithm significantly improves the EU evaluation over the existing methods.

Related papers

• On the Identifiability of Sparse ICA without Assuming Non-Gaussianity [20.333908367541895]
  We develop an identifiability theory that relies on second-order statistics without imposing further preconditions on the distribution of sources. We propose two estimation methods based on second-order statistics and sparsity constraint.
  arXiv  Detail & Related papers  (2024-08-19T18:51:42Z)
• Model-Based Epistemic Variance of Values for Risk-Aware Policy Optimization [59.758009422067]
  We consider the problem of quantifying uncertainty over expected cumulative rewards in model-based reinforcement learning. We propose a new uncertainty Bellman equation (UBE) whose solution converges to the true posterior variance over values. We introduce a general-purpose policy optimization algorithm, Q-Uncertainty Soft Actor-Critic (QU-SAC) that can be applied for either risk-seeking or risk-averse policy optimization.
  arXiv  Detail & Related papers  (2023-12-07T15:55:58Z)
• Selective Nonparametric Regression via Testing [54.20569354303575]
  We develop an abstention procedure via testing the hypothesis on the value of the conditional variance at a given point. Unlike existing methods, the proposed one allows to account not only for the value of the variance itself but also for the uncertainty of the corresponding variance predictor.
  arXiv  Detail & Related papers  (2023-09-28T13:04:11Z)
• Invariant Probabilistic Prediction [45.90606906307022]
  We show that arbitrary distribution shifts do not, in general, admit invariant and robust probabilistic predictions. We propose a method to yield invariant probabilistic predictions, called IPP, and study the consistency of the underlying parameters.
  arXiv  Detail & Related papers  (2023-09-18T18:50:24Z)
• Adversarial robustness of amortized Bayesian inference [3.308743964406687]
  Amortized Bayesian inference is to initially invest computational cost in training an inference network on simulated data. We show that almost unrecognizable, targeted perturbations of the observations can lead to drastic changes in the predicted posterior and highly unrealistic posterior predictive samples. We propose a computationally efficient regularization scheme based on penalizing the Fisher information of the conditional density estimator.
  arXiv  Detail & Related papers  (2023-05-24T10:18:45Z)
• Non-Linear Spectral Dimensionality Reduction Under Uncertainty [107.01839211235583]
  We propose a new dimensionality reduction framework, called NGEU, which leverages uncertainty information and directly extends several traditional approaches. We show that the proposed NGEU formulation exhibits a global closed-form solution, and we analyze, based on the Rademacher complexity, how the underlying uncertainties theoretically affect the generalization ability of the framework.
  arXiv  Detail & Related papers  (2022-02-09T19:01:33Z)
• Causality and Generalizability: Identifiability and Learning Methods [0.0]
  This thesis contributes to the research areas concerning the estimation of causal effects, causal structure learning, and distributionally robust prediction methods. We present novel and consistent linear and non-linear causal effects estimators in instrumental variable settings that employ data-dependent mean squared prediction error regularization. We propose a general framework for distributional robustness with respect to intervention-induced distributions.
  arXiv  Detail & Related papers  (2021-10-04T13:12:11Z)
• When in Doubt: Neural Non-Parametric Uncertainty Quantification for Epidemic Forecasting [70.54920804222031]
  Most existing forecasting models disregard uncertainty quantification, resulting in mis-calibrated predictions. Recent works in deep neural models for uncertainty-aware time-series forecasting also have several limitations. We model the forecasting task as a probabilistic generative process and propose a functional neural process model called EPIFNP.
  arXiv  Detail & Related papers  (2021-06-07T18:31:47Z)
• DEUP: Direct Epistemic Uncertainty Prediction [56.087230230128185]
  Epistemic uncertainty is part of out-of-sample prediction error due to the lack of knowledge of the learner. We propose a principled approach for directly estimating epistemic uncertainty by learning to predict generalization error and subtracting an estimate of aleatoric uncertainty.
  arXiv  Detail & Related papers  (2021-02-16T23:50:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.