Towards Reliable Simulation-based Inference

• URL: http://arxiv.org/abs/2603.08947v1
• Date: Mon, 09 Mar 2026 21:29:13 GMT
• Title: Towards Reliable Simulation-based Inference
• Abstract summary: This thesis focuses on statistical models that take the form of scientific simulators.<n>We show that performing statistical analyses with machine learning involves a degree of approximation.<n>We introduce balancing, a way to regularize machine learning models to reduce overconfidence and favor calibrated or underconfident approximations.
• Score: 1.1168121941015015
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scientific knowledge expands by observing the world, hypothesizing some theories about it, and testing them against collected data. When those theories take the form of statistical models, statistical analyses are involved in the process of testing and refining scientific hypotheses. In this thesis, we focus on statistical models that take the form of scientific simulators and provide background about how machine learning can be used for statistical analyses in this context. The first part of this thesis is about showing empirically that performing statistical analyses with machine learning involves a degree of approximation. Specifically, all statistical analyses involve a level of uncertainty in the conclusions drawn, and we show that approximations can lead to overconfident conclusions. We draw caution regarding such overconfident conclusions and introduce a criterion to diagnose overconfident approximations. In the second part, we introduce balancing, a way to regularize machine learning models to reduce overconfidence and favor calibrated or underconfident approximations. Balancing is first introduced for neural ratio estimation algorithms and then extended to other algorithms. Intuition about why balancing leads to less overconfident solutions is provided, and it is shown empirically that balanced algorithms are often either close to calibrated or underconfident. The third part shows that Bayesian neural networks can also be used to mitigate the overconfidence of approximations. Unlike balancing, no regularization is required, and this solution can then work with few training samples and, hence, computationally expensive simulators. To that end, a new Bayesian neural network prior tailored for simulation-based inference is developed, and empirical results show a reduction in overconfidence compared to similar solutions without Bayesian neural networks.

Related papers

• A Statistical Assessment of Amortized Inference Under Signal-to-Noise Variation and Distribution Shift [0.9590253747787195]
  Recent success of deep neural networks and foundation models has given rise to a new paradigm in statistical modeling.<n>In amortized inference, substantial computation is invested upfront to train a neural network that can produce approximate posterior or predictions.<n>Despite the growing popularity of amortized inference, its statistical interpretation and its role within Bayesian inference remain poorly understood.
  arXiv  Detail & Related papers  (2026-01-12T19:21:51Z)
• Calibrated and uncertain? Evaluating uncertainty estimates in binary classification models [0.0]
  Rigorous statistical methods underpin the validity of scientific discovery, especially in the natural sciences.<n>We use the unifying framework of approximate Bayesian inference combined with empirical tests on carefully created synthetic classification datasets.<n>We check if the algorithms produce uncertainty estimates which reflect commonly desired properties, such as being well calibrated and exhibiting an increase in uncertainty for out-of-distribution data points.
  arXiv  Detail & Related papers  (2025-08-15T13:17:32Z)
• Calibrating Neural Simulation-Based Inference with Differentiable Coverage Probability [50.44439018155837]
  We propose to include a calibration term directly into the training objective of the neural model. By introducing a relaxation of the classical formulation of calibration error we enable end-to-end backpropagation. It is directly applicable to existing computational pipelines allowing reliable black-box posterior inference.
  arXiv  Detail & Related papers  (2023-10-20T10:20:45Z)
• Prediction-Powered Inference [68.97619568620709]
  Prediction-powered inference is a framework for performing valid statistical inference when an experimental dataset is supplemented with predictions from a machine-learning system. The framework yields simple algorithms for computing provably valid confidence intervals for quantities such as means, quantiles, and linear and logistic regression coefficients. Prediction-powered inference could enable researchers to draw valid and more data-efficient conclusions using machine learning.
  arXiv  Detail & Related papers  (2023-01-23T18:59:28Z)
• Nonparametric likelihood-free inference with Jensen-Shannon divergence for simulator-based models with categorical output [1.4298334143083322]
  Likelihood-free inference for simulator-based statistical models has attracted a surge of interest, both in the machine learning and statistics communities. Here we derive a set of theoretical results to enable estimation, hypothesis testing and construction of confidence intervals for model parameters using computation properties of the Jensen-Shannon- divergence. Such approximation offers a rapid alternative to more-intensive approaches and can be attractive for diverse applications of simulator-based models.
  arXiv  Detail & Related papers  (2022-05-22T18:00:13Z)
• Causality and Generalizability: Identifiability and Learning Methods [0.0]
  This thesis contributes to the research areas concerning the estimation of causal effects, causal structure learning, and distributionally robust prediction methods. We present novel and consistent linear and non-linear causal effects estimators in instrumental variable settings that employ data-dependent mean squared prediction error regularization. We propose a general framework for distributional robustness with respect to intervention-induced distributions.
  arXiv  Detail & Related papers  (2021-10-04T13:12:11Z)
• Improving Uncertainty Calibration via Prior Augmented Data [56.88185136509654]
  Neural networks have proven successful at learning from complex data distributions by acting as universal function approximators. They are often overconfident in their predictions, which leads to inaccurate and miscalibrated probabilistic predictions. We propose a solution by seeking out regions of feature space where the model is unjustifiably overconfident, and conditionally raising the entropy of those predictions towards that of the prior distribution of the labels.
  arXiv  Detail & Related papers  (2021-02-22T07:02:37Z)
• Double Robust Representation Learning for Counterfactual Prediction [68.78210173955001]
  We propose a novel scalable method to learn double-robust representations for counterfactual predictions. We make robust and efficient counterfactual predictions for both individual and average treatment effects. The algorithm shows competitive performance with the state-of-the-art on real world and synthetic data.
  arXiv  Detail & Related papers  (2020-10-15T16:39:26Z)
• Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
  We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers. We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model. Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
  arXiv  Detail & Related papers  (2020-06-22T21:12:31Z)
• Balance-Subsampled Stable Prediction [55.13512328954456]
  We propose a novel balance-subsampled stable prediction (BSSP) algorithm based on the theory of fractional factorial design. A design-theoretic analysis shows that the proposed method can reduce the confounding effects among predictors induced by the distribution shift. Numerical experiments on both synthetic and real-world data sets demonstrate that our BSSP algorithm significantly outperforms the baseline methods for stable prediction across unknown test data.
  arXiv  Detail & Related papers  (2020-06-08T07:01:38Z)
• A comprehensive study on the prediction reliability of graph neural networks for virtual screening [0.0]
  We investigate the effects of model architectures, regularization methods, and loss functions on the prediction performance and reliability of classification results. Our result highlights that correct choice of regularization and inference methods is evidently important to achieve high success rate.
  arXiv  Detail & Related papers  (2020-03-17T10:13:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.