A powerful rank-based correction to multiple testing under positive
dependency
- URL: http://arxiv.org/abs/2311.10900v2
- Date: Thu, 25 Jan 2024 15:43:15 GMT
- Title: A powerful rank-based correction to multiple testing under positive
dependency
- Authors: Alexander Timans, Christoph-Nikolas Straehle, Kaspar Sakmann, Eric
Nalisnick
- Abstract summary: We develop a novel multiple hypothesis testing correction with family-wise error rate (FWER) control.
Our proposed algorithm $textttmax-rank$ is conceptually straight-forward, relying on the use of a $max$-operator in the rank domain of computed test statistics.
- Score: 48.098218835606055
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We develop a novel multiple hypothesis testing correction with family-wise
error rate (FWER) control that efficiently exploits positive dependencies
between potentially correlated statistical hypothesis tests. Our proposed
algorithm $\texttt{max-rank}$ is conceptually straight-forward, relying on the
use of a $\max$-operator in the rank domain of computed test statistics. We
compare our approach to the frequently employed Bonferroni correction,
theoretically and empirically demonstrating its superiority over Bonferroni in
the case of existing positive dependency, and its equivalence otherwise. Our
advantage over Bonferroni increases as the number of tests rises, and we
maintain high statistical power whilst ensuring FWER control. We specifically
frame our algorithm in the context of parallel permutation testing, a scenario
that arises in our primary application of conformal prediction, a recently
popularized approach for quantifying uncertainty in complex predictive
settings.
Related papers
- Non-Convex Robust Hypothesis Testing using Sinkhorn Uncertainty Sets [18.46110328123008]
We present a new framework to address the non-robust hypothesis testing problem.
The goal is to seek the optimal detector that minimizes the maximum numerical risk.
arXiv Detail & Related papers (2024-03-21T20:29:43Z) - Precise Error Rates for Computationally Efficient Testing [75.63895690909241]
We revisit the question of simple-versus-simple hypothesis testing with an eye towards computational complexity.
An existing test based on linear spectral statistics achieves the best possible tradeoff curve between type I and type II error rates.
arXiv Detail & Related papers (2023-11-01T04:41:16Z) - Advancing Counterfactual Inference through Nonlinear Quantile Regression [77.28323341329461]
We propose a framework for efficient and effective counterfactual inference implemented with neural networks.
The proposed approach enhances the capacity to generalize estimated counterfactual outcomes to unseen data.
Empirical results conducted on multiple datasets offer compelling support for our theoretical assertions.
arXiv Detail & Related papers (2023-06-09T08:30:51Z) - Sequential Predictive Two-Sample and Independence Testing [114.4130718687858]
We study the problems of sequential nonparametric two-sample and independence testing.
We build upon the principle of (nonparametric) testing by betting.
arXiv Detail & Related papers (2023-04-29T01:30:33Z) - A Semi-Bayesian Nonparametric Estimator of the Maximum Mean Discrepancy
Measure: Applications in Goodness-of-Fit Testing and Generative Adversarial
Networks [3.623570119514559]
We propose a semi-Bayesian nonparametric (semi-BNP) procedure for the goodness-of-fit (GOF) test.
Our method introduces a novel Bayesian estimator for the maximum mean discrepancy (MMD) measure.
We demonstrate that our proposed test outperforms frequentist MMD-based methods by achieving a lower false rejection and acceptance rate of the null hypothesis.
arXiv Detail & Related papers (2023-03-05T10:36:21Z) - Sequential Kernelized Independence Testing [101.22966794822084]
We design sequential kernelized independence tests inspired by kernelized dependence measures.
We demonstrate the power of our approaches on both simulated and real data.
arXiv Detail & Related papers (2022-12-14T18:08:42Z) - Doubly Robust Kernel Statistics for Testing Distributional Treatment
Effects [18.791409397894835]
We build upon a previously introduced framework, Counterfactual Mean Embeddings, for representing causal distributions within Reproducing Kernel Hilbert Spaces (RKHS)
These improved estimators are inspired by doubly robust estimators of the causal mean, using a similar form within the kernel space.
This leads to new permutation based tests for distributional causal effects, using the estimators we propose as tests statistics.
arXiv Detail & Related papers (2022-12-09T15:32:19Z) - Sequential Permutation Testing of Random Forest Variable Importance
Measures [68.8204255655161]
It is proposed here to use sequential permutation tests and sequential p-value estimation to reduce the high computational costs associated with conventional permutation tests.
The results of simulation studies confirm that the theoretical properties of the sequential tests apply.
The numerical stability of the methods is investigated in two additional application studies.
arXiv Detail & Related papers (2022-06-02T20:16:50Z) - Addressing Maximization Bias in Reinforcement Learning with Two-Sample
Testing [0.0]
Overestimation bias is a known threat to value-based reinforcement-learning algorithms.
We propose an estimator that flexibly interpolates between over- and underestimation by adjusting the significance level of the underlying hypothesis tests.
A generalization, termed $K$-Estimator (KE), obeys the same bias and variance bounds as the TE while relying on a nearly arbitrary kernel function.
arXiv Detail & Related papers (2022-01-20T09:22:43Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.