Expected Information Gain Estimation via Density Approximations: Sample Allocation and Dimension Reduction

• URL: http://arxiv.org/abs/2411.08390v1
• Date: Wed, 13 Nov 2024 07:22:50 GMT
• Title: Expected Information Gain Estimation via Density Approximations: Sample Allocation and Dimension Reduction
• Authors: Fengyi Li, Ricardo Baptista, Youssef Marzouk,
• Abstract summary: We formulate flexible transport-based schemes for EIG estimation in general nonlinear/non-Gaussian settings. We show that with this optimal sample allocation, the MSE of the resulting EIG estimator converges more quickly than that of a standard nested Monte Carlo scheme. We then address the estimation of EIG in high dimensions, by deriving gradient-based upper bounds on the mutual information lost by projecting the parameters and/or observations to lower-dimensional subspaces.
• Score: 0.40964539027092906
• License:
• Abstract: Computing expected information gain (EIG) from prior to posterior (equivalently, mutual information between candidate observations and model parameters or other quantities of interest) is a fundamental challenge in Bayesian optimal experimental design. We formulate flexible transport-based schemes for EIG estimation in general nonlinear/non-Gaussian settings, compatible with both standard and implicit Bayesian models. These schemes are representative of two-stage methods for estimating or bounding EIG using marginal and conditional density estimates. In this setting, we analyze the optimal allocation of samples between training (density estimation) and approximation of the outer prior expectation. We show that with this optimal sample allocation, the MSE of the resulting EIG estimator converges more quickly than that of a standard nested Monte Carlo scheme. We then address the estimation of EIG in high dimensions, by deriving gradient-based upper bounds on the mutual information lost by projecting the parameters and/or observations to lower-dimensional subspaces. Minimizing these upper bounds yields projectors and hence low-dimensional EIG approximations that outperform approximations obtained via other linear dimension reduction schemes. Numerical experiments on a PDE-constrained Bayesian inverse problem also illustrate a favorable trade-off between dimension truncation and the modeling of non-Gaussianity, when estimating EIG from finite samples in high dimensions.

Related papers

• Quasi-Bayes meets Vines [2.3124143670964448]
  We propose a different way to extend Quasi-Bayesian prediction to high dimensions through the use of Sklar's theorem. We show that our proposed Quasi-Bayesian Vine (QB-Vine) is a fully non-parametric density estimator with emphan analytical form.
  arXiv  Detail & Related papers  (2024-06-18T16:31:02Z)
• Differentially Private Optimization with Sparse Gradients [60.853074897282625]
  We study differentially private (DP) optimization problems under sparsity of individual gradients. Building on this, we obtain pure- and approximate-DP algorithms with almost optimal rates for convex optimization with sparse gradients.
  arXiv  Detail & Related papers  (2024-04-16T20:01:10Z)
• Diffusion posterior sampling for simulation-based inference in tall data settings [53.17563688225137]
  Simulation-based inference ( SBI) is capable of approximating the posterior distribution that relates input parameters to a given observation. In this work, we consider a tall data extension in which multiple observations are available to better infer the parameters of the model. We compare our method to recently proposed competing approaches on various numerical experiments and demonstrate its superiority in terms of numerical stability and computational cost.
  arXiv  Detail & Related papers  (2024-04-11T09:23:36Z)
• Variational Bayesian Optimal Experimental Design with Normalizing Flows [0.837622912636323]
  Variational OED estimates a lower bound of the EIG without likelihood evaluations. We introduce the use of normalizing flows for representing variational distributions in vOED. We show that a composition of 4--5 layers is able to achieve lower EIG estimation bias.
  arXiv  Detail & Related papers  (2024-04-08T14:44:21Z)
• Goal-Oriented Bayesian Optimal Experimental Design for Nonlinear Models using Markov Chain Monte Carlo [0.0]
  We present a computational framework of predictive goal-oriented OED (GO-OED) suitable for nonlinear observation and prediction models. GO-OED seeks the experimental design providing the greatest EIG on the QoIs. We demonstrate the effectiveness of the overall nonlinear GO-OED method, and illustrate its differences versus conventional non-GO-OED.
  arXiv  Detail & Related papers  (2024-03-26T19:49:58Z)
• Scalable method for Bayesian experimental design without integrating over posterior distribution [0.0]
  We address the computational efficiency in solving the A-optimal Bayesian design of experiments problems. A-optimality is a widely used and easy-to-interpret criterion for Bayesian experimental design. This study presents a novel likelihood-free approach to the A-optimal experimental design.
  arXiv  Detail & Related papers  (2023-06-30T12:40:43Z)
• Optimization of Annealed Importance Sampling Hyperparameters [77.34726150561087]
  Annealed Importance Sampling (AIS) is a popular algorithm used to estimates the intractable marginal likelihood of deep generative models. We present a parameteric AIS process with flexible intermediary distributions and optimize the bridging distributions to use fewer number of steps for sampling. We assess the performance of our optimized AIS for marginal likelihood estimation of deep generative models and compare it to other estimators.
  arXiv  Detail & Related papers  (2022-09-27T07:58:25Z)
• Robust Expected Information Gain for Optimal Bayesian Experimental Design Using Ambiguity Sets [0.0]
  We define and analyze emphrobust expected information gain (REIG) REIG is a modification of the objective in EIG by minimizing an affine relaxation of EIG over an ambiguity set of perturbed distributions. We show that, when combined with a sampling-based approach to estimating EIG, REIG corresponds to a log-sum-exp' stabilization of the samples used to estimate EIG.
  arXiv  Detail & Related papers  (2022-05-20T01:07:41Z)
• Learning to Estimate Without Bias [57.82628598276623]
  Gauss theorem states that the weighted least squares estimator is a linear minimum variance unbiased estimation (MVUE) in linear models. In this paper, we take a first step towards extending this result to non linear settings via deep learning with bias constraints. A second motivation to BCE is in applications where multiple estimates of the same unknown are averaged for improved performance.
  arXiv  Detail & Related papers  (2021-10-24T10:23:51Z)
• Efficient Semi-Implicit Variational Inference [65.07058307271329]
  We propose an efficient and scalable semi-implicit extrapolational (SIVI) Our method maps SIVI's evidence to a rigorous inference of lower gradient values.
  arXiv  Detail & Related papers  (2021-01-15T11:39:09Z)
• On the Convergence Rate of Projected Gradient Descent for a Back-Projection based Objective [58.33065918353532]
  We consider a back-projection based fidelity term as an alternative to the common least squares (LS) We show that using the BP term, rather than the LS term, requires fewer iterations of optimization algorithms.
  arXiv  Detail & Related papers  (2020-05-03T00:58:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.