Safe Testing

• URL: http://arxiv.org/abs/1906.07801v5
• Date: Fri, 10 Mar 2023 13:14:45 GMT
• Title: Safe Testing
• Authors: Peter Gr\"unwald, Rianne de Heide, and Wouter Koolen
• Abstract summary: We develop the theory of hypothesis testing based on the e-value. Tests based on e-values are safe, i.e. they preserve Type-I error guarantees.
• Score: 0.9634859579172255
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We develop the theory of hypothesis testing based on the e-value, a notion of evidence that, unlike the p-value, allows for effortlessly combining results from several studies in the common scenario where the decision to perform a new study may depend on previous outcomes. Tests based on e-values are safe, i.e. they preserve Type-I error guarantees, under such optional continuation. We define growth-rate optimality (GRO) as an analogue of power in an optional continuation context, and we show how to construct GRO e-variables for general testing problems with composite null and alternative, emphasizing models with nuisance parameters. GRO e-values take the form of Bayes factors with special priors. We illustrate the theory using several classic examples including a one-sample safe t-test and the 2 x 2 contingency table. Sharing Fisherian, Neymanian and Jeffreys-Bayesian interpretations, e-values may provide a methodology acceptable to adherents of all three schools.

Related papers

• Conformal Generative Modeling with Improved Sample Efficiency through Sequential Greedy Filtering [55.15192437680943]
  Generative models lack rigorous statistical guarantees for their outputs. We propose a sequential conformal prediction method producing prediction sets that satisfy a rigorous statistical guarantee. This guarantee states that with high probability, the prediction sets contain at least one admissible (or valid) example.
  arXiv  Detail & Related papers  (2024-10-02T15:26:52Z)
• Selective Nonparametric Regression via Testing [54.20569354303575]
  We develop an abstention procedure via testing the hypothesis on the value of the conditional variance at a given point. Unlike existing methods, the proposed one allows to account not only for the value of the variance itself but also for the uncertainty of the corresponding variance predictor.
  arXiv  Detail & Related papers  (2023-09-28T13:04:11Z)
• Sequential Permutation Testing of Random Forest Variable Importance Measures [68.8204255655161]
  It is proposed here to use sequential permutation tests and sequential p-value estimation to reduce the high computational costs associated with conventional permutation tests. The results of simulation studies confirm that the theoretical properties of the sequential tests apply. The numerical stability of the methods is investigated in two additional application studies.
  arXiv  Detail & Related papers  (2022-06-02T20:16:50Z)
• Regularizing Variational Autoencoder with Diversity and Uncertainty Awareness [61.827054365139645]
  Variational Autoencoder (VAE) approximates the posterior of latent variables based on amortized variational inference. We propose an alternative model, DU-VAE, for learning a more Diverse and less Uncertain latent space.
  arXiv  Detail & Related papers  (2021-10-24T07:58:13Z)
• Safe Tests and Always-Valid Confidence Intervals for contingency tables and beyond [69.25055322530058]
  We develop E variables for testing whether two data streams come from the same source or not. These E variables lead to tests that remain safe, under flexible sampling scenarios such as optional stopping and continuation.
  arXiv  Detail & Related papers  (2021-06-04T20:12:13Z)
• On Model Identification and Out-of-Sample Prediction of Principal Component Regression: Applications to Synthetic Controls [20.96904429337912]
  We analyze principal component regression (PCR) in a high-dimensional error-in-variables setting with fixed design. We establish non-asymptotic out-of-sample prediction guarantees that improve upon the best known rates.
  arXiv  Detail & Related papers  (2020-10-27T17:07:36Z)
• Achieving Equalized Odds by Resampling Sensitive Attributes [13.114114427206678]
  We present a flexible framework for learning predictive models that approximately satisfy the equalized odds notion of fairness. This differentiable functional is used as a penalty driving the model parameters towards equalized odds. We develop a formal hypothesis test to detect whether a prediction rule violates this property, the first such test in the literature.
  arXiv  Detail & Related papers  (2020-06-08T00:18:34Z)
• Double Generative Adversarial Networks for Conditional Independence Testing [8.359770027722275]
  High-dimensional conditional independence testing is a key building block in statistics and machine learning. We propose an inferential procedure based on double generative adversarial networks (GANs)
  arXiv  Detail & Related papers  (2020-06-03T16:14:15Z)
• Decision-Making with Auto-Encoding Variational Bayes [71.44735417472043]
  We show that a posterior approximation distinct from the variational distribution should be used for making decisions. Motivated by these theoretical results, we propose learning several approximate proposals for the best model. In addition to toy examples, we present a full-fledged case study of single-cell RNA sequencing.
  arXiv  Detail & Related papers  (2020-02-17T19:23:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.