Repulsive Ensembles for Bayesian Inference in Physics-informed Neural Networks

• URL: http://arxiv.org/abs/2505.17308v1
• Date: Thu, 22 May 2025 21:58:40 GMT
• Title: Repulsive Ensembles for Bayesian Inference in Physics-informed Neural Networks
• Authors: Philipp Pilar, Markus Heinonen, Niklas Wahlström,
• Abstract summary: We consider the inverse problem and employ repulsive ensembles of PINNs for obtaining uncertainty estimates.<n>The repulsive ensemble produces significantly more accurate uncertainty estimates and exhibits higher sample diversity.
• Score: 10.075911116030621
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Physics-informed neural networks (PINNs) have proven an effective tool for solving differential equations, in particular when considering non-standard or ill-posed settings. When inferring solutions and parameters of the differential equation from data, uncertainty estimates are preferable to point estimates, as they give an idea about the accuracy of the solution. In this work, we consider the inverse problem and employ repulsive ensembles of PINNs (RE-PINN) for obtaining such estimates. The repulsion is implemented by adding a particular repulsive term to the loss function, which has the property that the ensemble predictions correspond to the true Bayesian posterior in the limit of infinite ensemble members. Where possible, we compare the ensemble predictions to Monte Carlo baselines. Whereas the standard ensemble tends to collapse to maximum-a-posteriori solutions, the repulsive ensemble produces significantly more accurate uncertainty estimates and exhibits higher sample diversity.

Related papers

• Improved Uncertainty Quantification in Physics-Informed Neural Networks Using Error Bounds and Solution Bundles [2.066173485843472]
  We train Bayesian Neural Networks that provide uncertainties over the solutions to differential equation systems provided by PINNs.<n>We use available error bounds over PINNs to formulate a heteroscedastic variance that improves the uncertainty estimation.
  arXiv  Detail & Related papers  (2025-05-09T22:40:39Z)
• Confidence Interval Construction and Conditional Variance Estimation with Dense ReLU Networks [11.218066045459778]
  This paper addresses the problems of conditional variance estimation and confidence interval construction in nonparametric regression using dense networks with the Rectified Linear Unit (ReLU) activation function.<n>We present a residual-based framework for conditional variance estimation, deriving nonasymptotic bounds for variance estimation under both heteroscedastic and homoscedastic settings.<n>We develop a ReLU network based robust bootstrap procedure for constructing confidence intervals for the true mean that comes with a theoretical guarantee on the coverage, providing a significant advancement in uncertainty quantification and the construction of reliable confidence intervals in deep learning settings.
  arXiv  Detail & Related papers  (2024-12-29T05:17:58Z)
• Semiparametric conformal prediction [79.6147286161434]
  We construct a conformal prediction set accounting for the joint correlation structure of the vector-valued non-conformity scores.<n>We flexibly estimate the joint cumulative distribution function (CDF) of the scores.<n>Our method yields desired coverage and competitive efficiency on a range of real-world regression problems.
  arXiv  Detail & Related papers  (2024-11-04T14:29:02Z)
• Multivariate root-n-consistent smoothing parameter free matching estimators and estimators of inverse density weighted expectations [51.000851088730684]
  We develop novel modifications of nearest-neighbor and matching estimators which converge at the parametric $sqrt n $-rate.<n>We stress that our estimators do not involve nonparametric function estimators and in particular do not rely on sample-size dependent parameters smoothing.
  arXiv  Detail & Related papers  (2024-07-11T13:28:34Z)
• Likelihood Ratio Confidence Sets for Sequential Decision Making [51.66638486226482]
  We revisit the likelihood-based inference principle and propose to use likelihood ratios to construct valid confidence sequences. Our method is especially suitable for problems with well-specified likelihoods. We show how to provably choose the best sequence of estimators and shed light on connections to online convex optimization.
  arXiv  Detail & Related papers  (2023-11-08T00:10:21Z)
• Structured Radial Basis Function Network: Modelling Diversity for Multiple Hypotheses Prediction [51.82628081279621]
  Multi-modal regression is important in forecasting nonstationary processes or with a complex mixture of distributions. A Structured Radial Basis Function Network is presented as an ensemble of multiple hypotheses predictors for regression problems. It is proved that this structured model can efficiently interpolate this tessellation and approximate the multiple hypotheses target distribution.
  arXiv  Detail & Related papers  (2023-09-02T01:27:53Z)
• Nonparametric Quantile Regression: Non-Crossing Constraints and Conformal Prediction [2.654399717608053]
  We propose a nonparametric quantile regression method using deep neural networks with a rectified linear unit penalty function to avoid quantile crossing. We establish non-asymptotic upper bounds for the excess risk of the proposed nonparametric quantile regression function estimators. Numerical experiments including simulation studies and a real data example are conducted to demonstrate the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2022-10-18T20:59:48Z)
• Jensen-Shannon Divergence Based Novel Loss Functions for Bayesian Neural Networks [2.4554686192257424]
  We formulate a novel loss function for BNNs based on a new modification to the generalized Jensen-Shannon (JS) divergence, which is bounded.<n>We find that the JS divergence-based variational inference is intractable, and hence employed a constrained optimization framework to formulate these losses.<n>Our theoretical analysis and empirical experiments on multiple regression and classification data sets suggest that the proposed losses perform better than the KL divergence-based loss, especially when the data sets are noisy or biased.
  arXiv  Detail & Related papers  (2022-09-23T01:47:09Z)
• Nonparametric Score Estimators [49.42469547970041]
  Estimating the score from a set of samples generated by an unknown distribution is a fundamental task in inference and learning of probabilistic models. We provide a unifying view of these estimators under the framework of regularized nonparametric regression. We propose score estimators based on iterative regularization that enjoy computational benefits from curl-free kernels and fast convergence.
  arXiv  Detail & Related papers  (2020-05-20T15:01:03Z)
• Bayesian Neural Networks With Maximum Mean Discrepancy Regularization [13.97417198693205]
  We show that our BNNs achieve higher accuracy on multiple benchmarks, including several image classification tasks. We also provide a new formulation for estimating the uncertainty on a given prediction, showing it performs in a more robust fashion against adversarial attacks.
  arXiv  Detail & Related papers  (2020-03-02T14:54:48Z)
• Being Bayesian, Even Just a Bit, Fixes Overconfidence in ReLU Networks [65.24701908364383]
  We show that a sufficient condition for a uncertainty on a ReLU network is "to be a bit Bayesian calibrated" We further validate these findings empirically via various standard experiments using common deep ReLU networks and Laplace approximations.
  arXiv  Detail & Related papers  (2020-02-24T08:52:06Z)
• An Equivalence between Bayesian Priors and Penalties in Variational Inference [8.45602005745865]
  In machine learning, it is common to optimize the parameters of a probabilistic model, modulated by an ad hoc regularization term that penalizes some values of the parameters. We fully characterize the regularizers that can arise according to this procedure, and provide a systematic way to compute the prior corresponding to a given penalty.
  arXiv  Detail & Related papers  (2020-02-01T09:48:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.