Likelihood-Free Frequentist Inference: Bridging Classical Statistics and Machine Learning for Reliable Simulator-Based Inference

• URL: http://arxiv.org/abs/2107.03920v9
• Date: Wed, 09 Oct 2024 16:47:08 GMT
• Title: Likelihood-Free Frequentist Inference: Bridging Classical Statistics and Machine Learning for Reliable Simulator-Based Inference
• Authors: Niccolò Dalmasso, Luca Masserano, David Zhao, Rafael Izbicki, Ann B. Lee,
• Abstract summary: We propose a modular inference framework that bridges classical statistics and modern machine learning. We refer to this framework as likelihood-free frequentist inference (LF2I)
• Score: 4.525512100042707
• License:
• Abstract: Many areas of science rely on simulators that implicitly encode intractable likelihood functions of complex systems. Classical statistical methods are poorly suited for these so-called likelihood-free inference (LFI) settings, especially outside asymptotic and low-dimensional regimes. At the same time, popular LFI methods - such as Approximate Bayesian Computation or more recent machine learning techniques - do not necessarily lead to valid scientific inference because they do not guarantee confidence sets with nominal coverage in general settings. In addition, LFI currently lacks practical diagnostic tools to check the actual coverage of computed confidence sets across the entire parameter space. In this work, we propose a modular inference framework that bridges classical statistics and modern machine learning to provide (i) a practical approach for constructing confidence sets with near finite-sample validity at any value of the unknown parameters, and (ii) interpretable diagnostics for estimating empirical coverage across the entire parameter space. We refer to this framework as likelihood-free frequentist inference (LF2I). Any method that defines a test statistic can leverage LF2I to create valid confidence sets and diagnostics without costly Monte Carlo or bootstrap samples at fixed parameter settings. We study two likelihood-based test statistics (ACORE and BFF) and demonstrate their performance on high-dimensional complex data. Code is available at https://github.com/lee-group-cmu/lf2i.

Related papers

• Statistical Inference for Temporal Difference Learning with Linear Function Approximation [62.69448336714418]
  Temporal Difference (TD) learning, arguably the most widely used for policy evaluation, serves as a natural framework for this purpose. In this paper, we study the consistency properties of TD learning with Polyak-Ruppert averaging and linear function approximation, and obtain three significant improvements over existing results.
  arXiv  Detail & Related papers  (2024-10-21T15:34:44Z)
• Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
  Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode. We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
  arXiv  Detail & Related papers  (2024-06-05T16:35:30Z)
• Online non-parametric likelihood-ratio estimation by Pearson-divergence functional minimization [55.98760097296213]
  We introduce a new framework for online non-parametric LRE (OLRE) for the setting where pairs of iid observations $(x_t sim p, x'_t sim q)$ are observed over time. We provide theoretical guarantees for the performance of the OLRE method along with empirical validation in synthetic experiments.
  arXiv  Detail & Related papers  (2023-11-03T13:20:11Z)
• Overlapping Batch Confidence Intervals on Statistical Functionals Constructed from Time Series: Application to Quantiles, Optimization, and Estimation [5.068678962285631]
  We propose a confidence interval procedure for statistical functionals constructed using data from a stationary time series. The OBx limits, certain functionals of the Wiener process parameterized by the size of the batches and the extent of their overlap, form the essential machinery for characterizing dependence.
  arXiv  Detail & Related papers  (2023-07-17T16:21:48Z)
• Fast Entropy-Based Methods of Word-Level Confidence Estimation for End-To-End Automatic Speech Recognition [86.21889574126878]
  We show how per-frame entropy values can be normalized and aggregated to obtain a confidence measure per unit and per word. We evaluate the proposed confidence measures on LibriSpeech test sets, and show that they are up to 2 and 4 times better than confidence estimation based on the maximum per-frame probability.
  arXiv  Detail & Related papers  (2022-12-16T20:27:40Z)
• Data-Driven Reachability analysis and Support set Estimation with Christoffel Functions [8.183446952097528]
  We present algorithms for estimating the forward reachable set of a dynamical system. The produced estimate is the sublevel set of a function called an empirical inverse Christoffel function. In addition to reachability analysis, the same approach can be applied to general problems of estimating the support of a random variable.
  arXiv  Detail & Related papers  (2021-12-18T20:25:34Z)
• Locally Valid and Discriminative Confidence Intervals for Deep Learning Models [37.57296694423751]
  Uncertainty information should be valid (guaranteeing coverage) and discriminative (more uncertain when the expected risk is high) Most existing Bayesian methods lack frequentist coverage guarantees and usually affect model performance. We propose Locally Valid and Discriminative confidence intervals (LVD), a simple, efficient and lightweight method to construct discriminative confidence intervals (CIs) for almost any deep learning model.
  arXiv  Detail & Related papers  (2021-06-01T04:39:56Z)
• Extending the statistical software package Engine for Likelihood-Free Inference [0.0]
  This dissertation focuses on the implementation of the Robust optimisation Monte Carlo (ROMC) method in the software package Engine for Likelihood-Free Inference (ELFI) Our implementation provides a robust and efficient solution to a practitioner who wants to perform inference on a simulator-based model.
  arXiv  Detail & Related papers  (2020-11-08T13:22:37Z)
• Estimating Structural Target Functions using Machine Learning and Influence Functions [103.47897241856603]
  We propose a new framework for statistical machine learning of target functions arising as identifiable functionals from statistical models. This framework is problem- and model-agnostic and can be used to estimate a broad variety of target parameters of interest in applied statistics. We put particular focus on so-called coarsening at random/doubly robust problems with partially unobserved information.
  arXiv  Detail & Related papers  (2020-08-14T16:48:29Z)
• Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
  We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers. We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model. Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
  arXiv  Detail & Related papers  (2020-06-22T21:12:31Z)
• Confidence Sets and Hypothesis Testing in a Likelihood-Free Inference Setting [5.145741425164947]
  $texttACORE$ is a frequentist approach to LFI that first formulates the classical likelihood ratio test (LRT) as a parametrized classification problem. $texttACORE$ is based on the key observation that the statistic, the rejection probability of the test, and the coverage of the confidence set are conditional distribution functions.
  arXiv  Detail & Related papers  (2020-02-24T17:34:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.