Likelihood-Free Frequentist Inference: Bridging Classical Statistics and
Machine Learning for Reliable Simulator-Based Inference
- URL: http://arxiv.org/abs/2107.03920v8
- Date: Sun, 19 Nov 2023 22:13:06 GMT
- Title: Likelihood-Free Frequentist Inference: Bridging Classical Statistics and
Machine Learning for Reliable Simulator-Based Inference
- Authors: Niccol\`o Dalmasso, Luca Masserano, David Zhao, Rafael Izbicki, Ann B.
Lee
- Abstract summary: We propose a unified and modular inference framework that bridges classical statistics and modern machine learning.
We refer to the general framework as likelihood-free frequentist inference (LF2I)
- Score: 3.9927092855811983
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Many areas of science make extensive use of computer simulators that
implicitly encode intractable likelihood functions of complex systems.
Classical statistical methods are poorly suited for these so-called
likelihood-free inference (LFI) settings, especially outside asymptotic and
low-dimensional regimes. At the same time, traditional LFI methods - such as
Approximate Bayesian Computation or more recent machine learning techniques -
do not guarantee confidence sets with nominal coverage in general settings
(i.e., with high-dimensional data, finite sample sizes, and for any parameter
value). In addition, there are no diagnostic tools to check the empirical
coverage of confidence sets provided by such methods across the entire
parameter space. In this work, we propose a unified and modular inference
framework that bridges classical statistics and modern machine learning
providing (i) a practical approach to the Neyman construction of confidence
sets with frequentist finite-sample coverage for any value of the unknown
parameters; and (ii) interpretable diagnostics that estimate the empirical
coverage across the entire parameter space. We refer to the general framework
as likelihood-free frequentist inference (LF2I). Any method that defines a test
statistic can leverage LF2I to create valid confidence sets and diagnostics
without costly Monte Carlo samples at fixed parameter settings. We study the
power of two likelihood-based test statistics (ACORE and BFF) and demonstrate
their empirical performance on high-dimensional, complex data. Code is
available at https://github.com/lee-group-cmu/lf2i.
Related papers
- Cycles of Thought: Measuring LLM Confidence through Stable Explanations [53.15438489398938]
Large language models (LLMs) can reach and even surpass human-level accuracy on a variety of benchmarks, but their overconfidence in incorrect responses is still a well-documented failure mode.
We propose a framework for measuring an LLM's uncertainty with respect to the distribution of generated explanations for an answer.
arXiv Detail & Related papers (2024-06-05T16:35:30Z) - Overlapping Batch Confidence Intervals on Statistical Functionals
Constructed from Time Series: Application to Quantiles, Optimization, and
Estimation [5.068678962285631]
We propose a confidence interval procedure for statistical functionals constructed using data from a stationary time series.
The OBx limits, certain functionals of the Wiener process parameterized by the size of the batches and the extent of their overlap, form the essential machinery for characterizing dependence.
arXiv Detail & Related papers (2023-07-17T16:21:48Z) - The Normalized Cross Density Functional: A Framework to Quantify
Statistical Dependence for Random Processes [6.625320950808605]
We present a novel approach to measuring statistical dependence between two random processes (r.p.) using a positive-definite function called the Normalized Cross Density (NCD)
NCD is derived directly from the probability density functions of two r.p. and constructs a data-dependent Hilbert space, the Normalized Cross-Density Hilbert Space (NCD-HS)
We mathematically prove that FMCA learns the dominant eigenvalues and eigenfunctions of NCD directly from realizations.
arXiv Detail & Related papers (2022-12-09T02:12:41Z) - Validation Diagnostics for SBI algorithms based on Normalizing Flows [55.41644538483948]
This work proposes easy to interpret validation diagnostics for multi-dimensional conditional (posterior) density estimators based on NF.
It also offers theoretical guarantees based on results of local consistency.
This work should help the design of better specified models or drive the development of novel SBI-algorithms.
arXiv Detail & Related papers (2022-11-17T15:48:06Z) - Learning Summary Statistics for Bayesian Inference with Autoencoders [58.720142291102135]
We use the inner dimension of deep neural network based Autoencoders as summary statistics.
To create an incentive for the encoder to encode all the parameter-related information but not the noise, we give the decoder access to explicit or implicit information that has been used to generate the training data.
arXiv Detail & Related papers (2022-01-28T12:00:31Z) - Data-Driven Reachability analysis and Support set Estimation with
Christoffel Functions [8.183446952097528]
We present algorithms for estimating the forward reachable set of a dynamical system.
The produced estimate is the sublevel set of a function called an empirical inverse Christoffel function.
In addition to reachability analysis, the same approach can be applied to general problems of estimating the support of a random variable.
arXiv Detail & Related papers (2021-12-18T20:25:34Z) - Estimating Structural Target Functions using Machine Learning and
Influence Functions [103.47897241856603]
We propose a new framework for statistical machine learning of target functions arising as identifiable functionals from statistical models.
This framework is problem- and model-agnostic and can be used to estimate a broad variety of target parameters of interest in applied statistics.
We put particular focus on so-called coarsening at random/doubly robust problems with partially unobserved information.
arXiv Detail & Related papers (2020-08-14T16:48:29Z) - Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers.
We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model.
Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
arXiv Detail & Related papers (2020-06-22T21:12:31Z) - Double Generative Adversarial Networks for Conditional Independence
Testing [8.359770027722275]
High-dimensional conditional independence testing is a key building block in statistics and machine learning.
We propose an inferential procedure based on double generative adversarial networks (GANs)
arXiv Detail & Related papers (2020-06-03T16:14:15Z) - Machine learning for causal inference: on the use of cross-fit
estimators [77.34726150561087]
Doubly-robust cross-fit estimators have been proposed to yield better statistical properties.
We conducted a simulation study to assess the performance of several estimators for the average causal effect (ACE)
When used with machine learning, the doubly-robust cross-fit estimators substantially outperformed all of the other estimators in terms of bias, variance, and confidence interval coverage.
arXiv Detail & Related papers (2020-04-21T23:09:55Z) - Confidence Sets and Hypothesis Testing in a Likelihood-Free Inference
Setting [5.145741425164947]
$texttACORE$ is a frequentist approach to LFI that first formulates the classical likelihood ratio test (LRT) as a parametrized classification problem.
$texttACORE$ is based on the key observation that the statistic, the rejection probability of the test, and the coverage of the confidence set are conditional distribution functions.
arXiv Detail & Related papers (2020-02-24T17:34:49Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.