Online Control of the False Discovery Rate under "Decision Deadlines"

• URL: http://arxiv.org/abs/2110.01583v1
• Date: Mon, 4 Oct 2021 17:28:09 GMT
• Title: Online Control of the False Discovery Rate under "Decision Deadlines"
• Authors: Aaron Fisher
• Abstract summary: Online testing procedures aim to control the extent of false discoveries over a sequence of hypothesis tests. Our method controls the false discovery rate (FDR) at every stage of testing, as well as at adaptively chosen stopping times.
• Score: 1.4213973379473654
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Online testing procedures aim to control the extent of false discoveries over a sequence of hypothesis tests, allowing for the possibility that early-stage test results influence the choice of hypotheses to be tested in later stages. Typically, online methods assume that a permanent decision regarding the current test (reject or not reject) must be made before advancing to the next test. We instead assume that each hypothesis requires an immediate preliminary decision, but also allows us to update that decision until a preset deadline. Roughly speaking, this lets us apply a Benjamini-Hochberg-type procedure over a moving window of hypotheses, where the threshold parameters for upcoming tests can be determined based on preliminary results. Our method controls the false discovery rate (FDR) at every stage of testing, as well as at adaptively chosen stopping times. These results apply even under arbitrary p-value dependency structures.

Related papers

• Ranking by Lifts: A Cost-Benefit Approach to Large-Scale A/B Tests [0.0]
  A/B testers conducting large-scale tests want to be able to control false rejections of the null. This work develops a decision-theoretic framework for maximizing profits subject to false discovery rate (FDR) control.
  arXiv  Detail & Related papers  (2024-07-01T07:40:08Z)
• Mitigating LLM Hallucinations via Conformal Abstention [70.83870602967625]
  We develop a principled procedure for determining when a large language model should abstain from responding in a general domain. We leverage conformal prediction techniques to develop an abstention procedure that benefits from rigorous theoretical guarantees on the hallucination rate (error rate) Experimentally, our resulting conformal abstention method reliably bounds the hallucination rate on various closed-book, open-domain generative question answering datasets.
  arXiv  Detail & Related papers  (2024-04-04T11:32:03Z)
• Uncertainty-Calibrated Test-Time Model Adaptation without Forgetting [55.17761802332469]
  Test-time adaptation (TTA) seeks to tackle potential distribution shifts between training and test data by adapting a given model w.r.t. any test sample. Prior methods perform backpropagation for each test sample, resulting in unbearable optimization costs to many applications. We propose an Efficient Anti-Forgetting Test-Time Adaptation (EATA) method which develops an active sample selection criterion to identify reliable and non-redundant samples.
  arXiv  Detail & Related papers  (2024-03-18T05:49:45Z)
• Online multiple testing with e-values [37.0397290998274]
  A scientist wishes to make as many discoveries as possible while ensuring the number of false discoveries is controlled. Prior methods for FDR control in the online setting have focused on formulating algorithms when specific dependency structures are assumed to exist between the test statistics of each hypothesis. Our algorithm, e-LOND, provides FDR control under arbitrary, possibly unknown, dependence.
  arXiv  Detail & Related papers  (2023-11-10T22:14:47Z)
• Sequential Predictive Two-Sample and Independence Testing [114.4130718687858]
  We study the problems of sequential nonparametric two-sample and independence testing. We build upon the principle of (nonparametric) testing by betting.
  arXiv  Detail & Related papers  (2023-04-29T01:30:33Z)
• Sequential Kernelized Independence Testing [101.22966794822084]
  We design sequential kernelized independence tests inspired by kernelized dependence measures. We demonstrate the power of our approaches on both simulated and real data.
  arXiv  Detail & Related papers  (2022-12-14T18:08:42Z)
• Shortcomings of Top-Down Randomization-Based Sanity Checks for Evaluations of Deep Neural Network Explanations [67.40641255908443]
  We identify limitations of model-randomization-based sanity checks for the purpose of evaluating explanations. Top-down model randomization preserves scales of forward pass activations with high probability.
  arXiv  Detail & Related papers  (2022-11-22T18:52:38Z)
• Model-Free Sequential Testing for Conditional Independence via Testing by Betting [8.293345261434943]
  The proposed test allows researchers to analyze an incoming i.i.d. data stream with any arbitrary dependency structure. We allow the processing of data points online as soon as they arrive and stop data acquisition once significant results are detected.
  arXiv  Detail & Related papers  (2022-10-01T20:05:33Z)
• Private Sequential Hypothesis Testing for Statisticians: Privacy, Error Rates, and Sample Size [24.149533870085175]
  We study the sequential hypothesis testing problem under a slight variant of differential privacy, known as Renyi differential privacy. We present a new private algorithm based on Wald's Sequential Probability Ratio Test (SPRT) that also gives strong theoretical privacy guarantees.
  arXiv  Detail & Related papers  (2022-04-10T04:15:50Z)
• Noisy Adaptive Group Testing using Bayesian Sequential Experimental Design [63.48989885374238]
  When the infection prevalence of a disease is low, Dorfman showed 80 years ago that testing groups of people can prove more efficient than testing people individually. Our goal in this paper is to propose new group testing algorithms that can operate in a noisy setting.
  arXiv  Detail & Related papers  (2020-04-26T23:41:33Z)
• PAPRIKA: Private Online False Discovery Rate Control [27.698099204682105]
  We study False Discovery Rate (FDR) control in hypothesis testing under the constraint of differential privacy for the sample. We provide new private algorithms based on state-of-the-art results in non-private online FDR control.
  arXiv  Detail & Related papers  (2020-02-27T18:42:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.