Online multiple testing with e-values
- URL: http://arxiv.org/abs/2311.06412v1
- Date: Fri, 10 Nov 2023 22:14:47 GMT
- Title: Online multiple testing with e-values
- Authors: Ziyu Xu, Aaditya Ramdas
- Abstract summary: A scientist wishes to make as many discoveries as possible while ensuring the number of false discoveries is controlled.
Prior methods for FDR control in the online setting have focused on formulating algorithms when specific dependency structures are assumed to exist between the test statistics of each hypothesis.
Our algorithm, e-LOND, provides FDR control under arbitrary, possibly unknown, dependence.
- Score: 37.0397290998274
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: A scientist tests a continuous stream of hypotheses over time in the course
of her investigation -- she does not test a predetermined, fixed number of
hypotheses. The scientist wishes to make as many discoveries as possible while
ensuring the number of false discoveries is controlled -- a well recognized way
for accomplishing this is to control the false discovery rate (FDR). Prior
methods for FDR control in the online setting have focused on formulating
algorithms when specific dependency structures are assumed to exist between the
test statistics of each hypothesis. However, in practice, these dependencies
often cannot be known beforehand or tested after the fact. Our algorithm,
e-LOND, provides FDR control under arbitrary, possibly unknown, dependence. We
show that our method is more powerful than existing approaches to this problem
through simulations. We also formulate extensions of this algorithm to utilize
randomization for increased power, and for constructing confidence intervals in
online selective inference.
Related papers
- Mitigating LLM Hallucinations via Conformal Abstention [70.83870602967625]
We develop a principled procedure for determining when a large language model should abstain from responding in a general domain.
We leverage conformal prediction techniques to develop an abstention procedure that benefits from rigorous theoretical guarantees on the hallucination rate (error rate)
Experimentally, our resulting conformal abstention method reliably bounds the hallucination rate on various closed-book, open-domain generative question answering datasets.
arXiv Detail & Related papers (2024-04-04T11:32:03Z) - Sequential Predictive Two-Sample and Independence Testing [114.4130718687858]
We study the problems of sequential nonparametric two-sample and independence testing.
We build upon the principle of (nonparametric) testing by betting.
arXiv Detail & Related papers (2023-04-29T01:30:33Z) - Sequential Kernelized Independence Testing [101.22966794822084]
We design sequential kernelized independence tests inspired by kernelized dependence measures.
We demonstrate the power of our approaches on both simulated and real data.
arXiv Detail & Related papers (2022-12-14T18:08:42Z) - Validation Diagnostics for SBI algorithms based on Normalizing Flows [55.41644538483948]
This work proposes easy to interpret validation diagnostics for multi-dimensional conditional (posterior) density estimators based on NF.
It also offers theoretical guarantees based on results of local consistency.
This work should help the design of better specified models or drive the development of novel SBI-algorithms.
arXiv Detail & Related papers (2022-11-17T15:48:06Z) - Sequential Permutation Testing of Random Forest Variable Importance
Measures [68.8204255655161]
It is proposed here to use sequential permutation tests and sequential p-value estimation to reduce the high computational costs associated with conventional permutation tests.
The results of simulation studies confirm that the theoretical properties of the sequential tests apply.
The numerical stability of the methods is investigated in two additional application studies.
arXiv Detail & Related papers (2022-06-02T20:16:50Z) - SAFFRON and LORD Ensure Online Control of the False Discovery Rate Under
Positive Dependence [1.4213973379473654]
Some of the most popular online methods include alpha investing, LORD++ (hereafter, LORD), and SAFFRON.
These three methods have been shown to provide online control of the "modified" false discovery rate (mFDR)
Our work bolsters these results by showing that SAFFRON and LORD additionally ensure online control of the FDR under nonnegative dependence.
arXiv Detail & Related papers (2021-10-15T15:43:24Z) - Online Control of the False Discovery Rate under "Decision Deadlines" [1.4213973379473654]
Online testing procedures aim to control the extent of false discoveries over a sequence of hypothesis tests.
Our method controls the false discovery rate (FDR) at every stage of testing, as well as at adaptively chosen stopping times.
arXiv Detail & Related papers (2021-10-04T17:28:09Z) - Dynamic Algorithms for Online Multiple Testing [38.45810475976042]
We derive new algorithms for online multiple testing that provably control false discovery exceedance (FDX)
We demonstrate that our algorithms achieve higher power in a variety of synthetic experiments.
SupLORD is the first non-trivial algorithm, to our knowledge, that can control FDR at stopping times in the online setting.
arXiv Detail & Related papers (2020-10-26T23:41:54Z) - PAPRIKA: Private Online False Discovery Rate Control [27.698099204682105]
We study False Discovery Rate (FDR) control in hypothesis testing under the constraint of differential privacy for the sample.
We provide new private algorithms based on state-of-the-art results in non-private online FDR control.
arXiv Detail & Related papers (2020-02-27T18:42:23Z) - Independence Testing for Temporal Data [14.25244839642841]
A fundamental question is whether two time-series are related or not.
Existing approaches often have limitations, such as relying on parametric assumptions.
This paper introduces the temporal dependence statistic with block permutation to test independence between temporal data.
arXiv Detail & Related papers (2019-08-18T17:19:16Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.