Composite Goodness-of-fit Tests with Kernels
- URL: http://arxiv.org/abs/2111.10275v4
- Date: Tue, 27 Feb 2024 11:32:09 GMT
- Title: Composite Goodness-of-fit Tests with Kernels
- Authors: Oscar Key, Arthur Gretton, Fran\c{c}ois-Xavier Briol, Tamara Fernandez
- Abstract summary: We propose a kernel-based hypothesis tests for the challenging composite testing problem.
Our tests make use of minimum distance estimators based on the maximum mean discrepancy and the kernel Stein discrepancy.
As our main result, we show that we are able to estimate the parameter and conduct our test on the same data, while maintaining a correct test level.
- Score: 19.744607024807188
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Model misspecification can create significant challenges for the
implementation of probabilistic models, and this has led to development of a
range of robust methods which directly account for this issue. However, whether
these more involved methods are required will depend on whether the model is
really misspecified, and there is a lack of generally applicable methods to
answer this question. In this paper, we propose one such method. More
precisely, we propose kernel-based hypothesis tests for the challenging
composite testing problem, where we are interested in whether the data comes
from any distribution in some parametric family. Our tests make use of minimum
distance estimators based on the maximum mean discrepancy and the kernel Stein
discrepancy. They are widely applicable, including whenever the density of the
parametric model is known up to normalisation constant, or if the model takes
the form of a simulator. As our main result, we show that we are able to
estimate the parameter and conduct our test on the same data (without data
splitting), while maintaining a correct test level. Our approach is illustrated
on a range of problems, including testing for goodness-of-fit of an
unnormalised non-parametric density model, and an intractable generative model
of a biological cellular network.
Related papers
- On the Robustness of Kernel Goodness-of-Fit Tests [5.959410850280868]
We show that existing kernel goodness-of-fit tests are not robust according to common notions of robustness.
We propose the first robust kernel goodness-of-fit test which resolves this open problem using kernel Stein discrepancy balls.
arXiv Detail & Related papers (2024-08-11T19:54:50Z) - Diffusion posterior sampling for simulation-based inference in tall data settings [53.17563688225137]
Simulation-based inference ( SBI) is capable of approximating the posterior distribution that relates input parameters to a given observation.
In this work, we consider a tall data extension in which multiple observations are available to better infer the parameters of the model.
We compare our method to recently proposed competing approaches on various numerical experiments and demonstrate its superiority in terms of numerical stability and computational cost.
arXiv Detail & Related papers (2024-04-11T09:23:36Z) - Test Set Sizing Via Random Matrix Theory [91.3755431537592]
This paper uses techniques from Random Matrix Theory to find the ideal training-testing data split for a simple linear regression.
It defines "ideal" as satisfying the integrity metric, i.e. the empirical model error is the actual measurement noise.
This paper is the first to solve for the training and test size for any model in a way that is truly optimal.
arXiv Detail & Related papers (2021-12-11T13:18:33Z) - Universal Inference Meets Random Projections: A Scalable Test for Log-concavity [30.073886309373226]
We present the first test of log-concavity that is provably valid in finite samples in any dimension.
We find that a random projections approach that converts the d-dimensional testing problem into many one-dimensional problems can yield high power.
arXiv Detail & Related papers (2021-11-17T17:34:44Z) - Training on Test Data with Bayesian Adaptation for Covariate Shift [96.3250517412545]
Deep neural networks often make inaccurate predictions with unreliable uncertainty estimates.
We derive a Bayesian model that provides for a well-defined relationship between unlabeled inputs under distributional shift and model parameters.
We show that our method improves both accuracy and uncertainty estimation.
arXiv Detail & Related papers (2021-09-27T01:09:08Z) - Evaluating State-of-the-Art Classification Models Against Bayes
Optimality [106.50867011164584]
We show that we can compute the exact Bayes error of generative models learned using normalizing flows.
We use our approach to conduct a thorough investigation of state-of-the-art classification models.
arXiv Detail & Related papers (2021-06-07T06:21:20Z) - Testing for Typicality with Respect to an Ensemble of Learned
Distributions [5.850572971372637]
One-sample approaches to the goodness-of-fit problem offer significant computational advantages for online testing.
The ability to correctly reject anomalous data in this setting hinges on the accuracy of the model of the base distribution.
Existing methods for the one-sample goodness-of-fit problem do not account for the fact that a model of the base distribution is learned.
We propose training an ensemble of density models, considering data to be anomalous if the data is anomalous with respect to any member of the ensemble.
arXiv Detail & Related papers (2020-11-11T19:47:46Z) - Good Classifiers are Abundant in the Interpolating Regime [64.72044662855612]
We develop a methodology to compute precisely the full distribution of test errors among interpolating classifiers.
We find that test errors tend to concentrate around a small typical value $varepsilon*$, which deviates substantially from the test error of worst-case interpolating model.
Our results show that the usual style of analysis in statistical learning theory may not be fine-grained enough to capture the good generalization performance observed in practice.
arXiv Detail & Related papers (2020-06-22T21:12:31Z) - A Causal Direction Test for Heterogeneous Populations [10.653162005300608]
Most causal models assume a single homogeneous population, an assumption that may fail to hold in many applications.
We show that when the homogeneity assumption is violated, causal models developed based on such assumption can fail to identify the correct causal direction.
We propose an adjustment to a commonly used causal direction test statistic by using a $k$-means type clustering algorithm.
arXiv Detail & Related papers (2020-06-08T18:59:14Z) - A Kernel Stein Test for Comparing Latent Variable Models [48.32146056855925]
We propose a kernel-based nonparametric test of relative goodness of fit, where the goal is to compare two models, both of which may have unobserved latent variables.
We show that our test significantly outperforms the relative Maximum Mean Discrepancy test, which is based on samples from the models and does not exploit the latent structure.
arXiv Detail & Related papers (2019-07-01T07:46:16Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.