On the Adversarial Robustness of Benjamini Hochberg

• URL: http://arxiv.org/abs/2501.03402v1
• Date: Mon, 06 Jan 2025 21:41:53 GMT
• Title: On the Adversarial Robustness of Benjamini Hochberg
• Authors: Louis L Chen, Roberto Szechtman, Matan Seri,
• Abstract summary: The Benjamini-Hochberg (BH) procedure is widely used to control the false detection rate (FDR) in multiple testing.<n>Considering this control could be relied upon in critical safety/security contexts, we investigate its adversarial robustness.
• Score: 0.06144680854063938
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The Benjamini-Hochberg (BH) procedure is widely used to control the false detection rate (FDR) in multiple testing. Applications of this control abound in drug discovery, forensics, anomaly detection, and, in particular, machine learning, ranging from nonparametric outlier detection to out-of-distribution detection and one-class classification methods. Considering this control could be relied upon in critical safety/security contexts, we investigate its adversarial robustness. More precisely, we study under what conditions BH does and does not exhibit adversarial robustness, we present a class of simple and easily implementable adversarial test-perturbation algorithms, and we perform computational experiments. With our algorithms, we demonstrate that there are conditions under which BH's control can be significantly broken with relatively few (even just one) test score perturbation(s), and provide non-asymptotic guarantees on the expected adversarial-adjustment to FDR. Our technical analysis involves a combinatorial reframing of the BH procedure as a ``balls into bins'' process, and drawing a connection to generalized ballot problems to facilitate an information-theoretic approach for deriving non-asymptotic lower bounds.

Related papers

• Beyond Raw Detection Scores: Markov-Informed Calibration for Boosting Machine-Generated Text Detection [105.14032334647932]
  Machine-generated texts (MGTs) pose risks such as disinformation and phishing, highlighting the need for reliable detection.<n> Metric-based methods, which extract statistically distinguishable features of MGTs, are often more practical than complex model-based methods that are prone to overfitting.<n>We propose a Markov-informed score calibration strategy that models two relationships of context detection scores that may aid calibration.
  arXiv  Detail & Related papers  (2026-02-08T16:06:12Z)
• Conformal novelty detection with false discovery rate control at the boundary [3.10490198369453]
  Conformal novelty detection is a classical machine learning task.<n>Recent work has shown that the BH procedure applied to conformal p-values controls the false discovery rate (FDR)
  arXiv  Detail & Related papers  (2026-01-06T00:02:03Z)
• Discovering Causal Relationships using Proxy Variables under Unmeasured Confounding [42.70985072862832]
  Inferring causal relationships between variable pairs in the observational study is crucial but challenging.<n>We develop a general nonparametric approach that accommodates both discrete and continuous settings for testing causal hypothesis under unmeasured confounders.<n>We demonstrate the effectiveness of our approach through extensive simulations and real-world data from the Intensive Care Data and World Values Survey.
  arXiv  Detail & Related papers  (2025-10-20T05:13:12Z)
• On the Adversarial Robustness of Learning-based Conformal Novelty Detection [10.58528988397402]
  We study the adversarial robustness of conformal novelty detection using AdaDetect.<n>Our results show that adversarial perturbations can significantly increase the FDR while maintaining high detection power.
  arXiv  Detail & Related papers  (2025-10-01T03:29:11Z)
• Asymptotically Optimal Linear Best Feasible Arm Identification with Fixed Budget [55.938644481736446]
  We introduce a novel algorithm for best feasible arm identification that guarantees an exponential decay in the error probability.<n>We validate our algorithm through comprehensive empirical evaluations across various problem instances with different levels of complexity.
  arXiv  Detail & Related papers  (2025-06-03T02:56:26Z)
• Conformal Prediction with Cellwise Outliers: A Detect-then-Impute Approach [4.328280329592151]
  Conformal prediction is a powerful tool for constructing prediction intervals for black-box models.<n>This paper introduces a novel framework called detect-then-impute conformal prediction.<n>We develop two practical algorithms, PDI-CP and JDI-CP, and provide a distribution-free coverage analysis.
  arXiv  Detail & Related papers  (2025-05-08T06:43:18Z)
• Verify when Uncertain: Beyond Self-Consistency in Black Box Hallucination Detection [25.176984317213858]
  Large Language Models (LLMs) suffer from hallucination problems, which hinder their reliability in sensitive applications. We propose a budget-friendly, two-stage detection algorithm that calls the verifier model only for a subset of cases.
  arXiv  Detail & Related papers  (2025-02-20T21:06:08Z)
• Scalable Similarity-Aware Test Suite Minimization with Reinforcement Learning [6.9290255098776425]
  TripRL produces a diverse reduced test suite with high test effectiveness. We show that TripRL's runtime scales linearly with the magnitude of the Multi-Criteria Test Suite Minimization problem.
  arXiv  Detail & Related papers  (2024-08-24T08:43:03Z)
• Best Arm Identification with Fixed Budget: A Large Deviation Perspective [54.305323903582845]
  We present sred, a truly adaptive algorithm that can reject arms in it any round based on the observed empirical gaps between the rewards of various arms. In particular, we present sred, a truly adaptive algorithm that can reject arms in it any round based on the observed empirical gaps between the rewards of various arms.
  arXiv  Detail & Related papers  (2023-12-19T13:17:43Z)
• B-BACN: Bayesian Boundary-Aware Convolutional Network for Crack Characterization [4.447467536572625]
  Uncertainty of crack detection is challenging due to various factors, such as measurement noises, signal processing, and model simplifications. A machine learning-based approach is proposed to quantify both uncertainty and aleatoric uncertainties concurrently. We introduce a Boundary-Aware Convolutional Network (B-BACN) that emphasizes uncertainty-aware boundary refinement to generate precise and reliable crack boundary detections.
  arXiv  Detail & Related papers  (2023-02-14T04:50:42Z)
• Adaptive novelty detection with false discovery rate guarantee [1.8249324194382757]
  We propose a flexible method to control the false discovery rate (FDR) on detected novelties in finite samples. Inspired by the multiple testing literature, we propose variants of AdaDetect that are adaptive to the proportion of nulls. The methods are illustrated on synthetic datasets and real-world datasets, including an application in astrophysics.
  arXiv  Detail & Related papers  (2022-08-13T17:14:55Z)
• Statistical and Computational Phase Transitions in Group Testing [73.55361918807883]
  We study the group testing problem where the goal is to identify a set of k infected individuals carrying a rare disease. We consider two different simple random procedures for assigning individuals tests.
  arXiv  Detail & Related papers  (2022-06-15T16:38:50Z)
• Mitigating the Mutual Error Amplification for Semi-Supervised Object Detection [92.52505195585925]
  We propose a Cross Teaching (CT) method, aiming to mitigate the mutual error amplification by introducing a rectification mechanism of pseudo labels. In contrast to existing mutual teaching methods that directly treat predictions from other detectors as pseudo labels, we propose the Label Rectification Module (LRM)
  arXiv  Detail & Related papers  (2022-01-26T03:34:57Z)
• Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control [67.52000805944924]
  Learn then Test (LTT) is a framework for calibrating machine learning models. Our main insight is to reframe the risk-control problem as multiple hypothesis testing. We use our framework to provide new calibration methods for several core machine learning tasks with detailed worked examples in computer vision.
  arXiv  Detail & Related papers  (2021-10-03T17:42:03Z)
• Unsupervised Anomaly Detection Ensembles using Item Response Theory [0.4640835690336652]
  We use Item Response Theory (IRT) to construct an unsupervised anomaly detection ensemble. IRT's latent trait lends itself to anomaly detection because the latent trait can be used to uncover the hidden ground truth. We demonstrate the effectiveness of the IRT ensemble on an extensive data repository.
  arXiv  Detail & Related papers  (2021-06-11T08:51:26Z)
• Probabilistic Ranking-Aware Ensembles for Enhanced Object Detections [50.096540945099704]
  We propose a novel ensemble called the Probabilistic Ranking Aware Ensemble (PRAE) that refines the confidence of bounding boxes from detectors. We also introduce a bandit approach to address the confidence imbalance problem caused by the need to deal with different numbers of boxes.
  arXiv  Detail & Related papers  (2021-05-07T09:37:06Z)
• Generalized Chernoff Sampling for Active Testing, Active Regression and Structured Bandit Algorithms [16.19565714525819]
  This paper studies active learning and best-arm identification in structured bandit settings. We obtain a novel sample bound complexity for Chernoff's original active testing procedure.
  arXiv  Detail & Related papers  (2020-12-15T03:44:18Z)
• The Simulator: Understanding Adaptive Sampling in the Moderate-Confidence Regime [52.38455827779212]
  We propose a novel technique for analyzing adaptive sampling called the em Simulator. We prove the first instance-based lower bounds the top-k problem which incorporate the appropriate log-factors. Our new analysis inspires a simple and near-optimal for the best-arm and top-k identification, the first em practical of its kind for the latter problem.
  arXiv  Detail & Related papers  (2017-02-16T23:42:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.