DSDE: Using Proportion Estimation to Improve Model Selection for Out-of-Distribution Detection

• URL: http://arxiv.org/abs/2411.01487v1
• Date: Sun, 03 Nov 2024 09:01:36 GMT
• Title: DSDE: Using Proportion Estimation to Improve Model Selection for Out-of-Distribution Detection
• Authors: Jingyao Geng, Yuan Zhang, Jiaqi Huang, Feng Xue, Falong Tan, Chuanlong Xie, Shumei Zhang,
• Abstract summary: Experimental results on CIFAR10 and CIFAR100 demonstrate the effectiveness of our approach in tackling OoD detection challenges. We name the proposed approach as DOS-Storey-based Detector Ensemble (DSDE)
• Score: 15.238164468992148
• License:
• Abstract: Model library is an effective tool for improving the performance of single-model Out-of-Distribution (OoD) detector, mainly through model selection and detector fusion. However, existing methods in the literature do not provide uncertainty quantification for model selection results. Additionally, the model ensemble process primarily focuses on controlling the True Positive Rate (TPR) while neglecting the False Positive Rate (FPR). In this paper, we emphasize the significance of the proportion of models in the library that identify the test sample as an OoD sample. This proportion holds crucial information and directly influences the error rate of OoD detection.To address this, we propose inverting the commonly-used sequential p-value strategies. We define the rejection region initially and then estimate the error rate. Furthermore, we introduce a novel perspective from change-point detection and propose an approach for proportion estimation with automatic hyperparameter selection. We name the proposed approach as DOS-Storey-based Detector Ensemble (DSDE). Experimental results on CIFAR10 and CIFAR100 demonstrate the effectiveness of our approach in tackling OoD detection challenges. Specifically, the CIFAR10 experiments show that DSDE reduces the FPR from 11.07% to 3.31% compared to the top-performing single-model detector.

Related papers

• GAIA: Delving into Gradient-based Attribution Abnormality for Out-of-distribution Detection [40.07502368794068]
  We offer an innovative perspective on quantifying the disparities between in-distribution (ID) and out-of-distribution (OOD) data. We introduce two forms of abnormalities for OOD detection: the zero-deflation abnormality and the channel-wise average abnormality. The effectiveness of GAIA is validated on both commonly utilized (CIFAR) and large-scale (ImageNet-1k) benchmarks.
  arXiv  Detail & Related papers  (2023-11-16T07:05:12Z)
• AdjointDPM: Adjoint Sensitivity Method for Gradient Backpropagation of Diffusion Probabilistic Models [103.41269503488546]
  Existing customization methods require access to multiple reference examples to align pre-trained diffusion probabilistic models with user-provided concepts. This paper aims to address the challenge of DPM customization when the only available supervision is a differentiable metric defined on the generated contents. We propose a novel method AdjointDPM, which first generates new samples from diffusion models by solving the corresponding probability-flow ODEs. It then uses the adjoint sensitivity method to backpropagate the gradients of the loss to the models' parameters.
  arXiv  Detail & Related papers  (2023-07-20T09:06:21Z)
• Boosting Out-of-Distribution Detection with Multiple Pre-trained Models [41.66566916581451]
  Post hoc detection utilizing pre-trained models has shown promising performance and can be scaled to large-scale problems. We propose a detection enhancement method by ensembling multiple detection decisions derived from a zoo of pre-trained models. Our method substantially improves the relative performance by 65.40% and 26.96% on the CIFAR10 and ImageNet benchmarks.
  arXiv  Detail & Related papers  (2022-12-24T12:11:38Z)
• Diffusion Denoising Process for Perceptron Bias in Out-of-distribution Detection [67.49587673594276]
  We introduce a new perceptron bias assumption that suggests discriminator models are more sensitive to certain features of the input, leading to the overconfidence problem. We demonstrate that the diffusion denoising process (DDP) of DMs serves as a novel form of asymmetric, which is well-suited to enhance the input and mitigate the overconfidence problem. Our experiments on CIFAR10, CIFAR100, and ImageNet show that our method outperforms SOTA approaches.
  arXiv  Detail & Related papers  (2022-11-21T08:45:08Z)
• Fake It Till You Make It: Near-Distribution Novelty Detection by Score-Based Generative Models [54.182955830194445]
  existing models either fail or face a dramatic drop under the so-called near-distribution" setting. We propose to exploit a score-based generative model to produce synthetic near-distribution anomalous data. Our method improves the near-distribution novelty detection by 6% and passes the state-of-the-art by 1% to 5% across nine novelty detection benchmarks.
  arXiv  Detail & Related papers  (2022-05-28T02:02:53Z)
• Efficient Test-Time Model Adaptation without Forgetting [60.36499845014649]
  Test-time adaptation seeks to tackle potential distribution shifts between training and testing data. We propose an active sample selection criterion to identify reliable and non-redundant samples. We also introduce a Fisher regularizer to constrain important model parameters from drastic changes.
  arXiv  Detail & Related papers  (2022-04-06T06:39:40Z)
• Error-based Knockoffs Inference for Controlled Feature Selection [49.99321384855201]
  We propose an error-based knockoff inference method by integrating the knockoff features, the error-based feature importance statistics, and the stepdown procedure together. The proposed inference procedure does not require specifying a regression model and can handle feature selection with theoretical guarantees.
  arXiv  Detail & Related papers  (2022-03-09T01:55:59Z)
• Adversarial Detection without Model Information [4.4890837610757695]
  We propose a model independent adversarial detection method using a simple energy function. We train a standalone detector independent of the underlying model, with sequential layer-wise training to increase the energy separation corresponding to natural and adversarial inputs. Our method achieves state-of-the-art detection performance (ROC-AUC > 0.9) across a wide range of gradient, score and decision-based adversarial attacks.
  arXiv  Detail & Related papers  (2022-02-09T04:38:16Z)
• Selecting Treatment Effects Models for Domain Adaptation Using Causal Knowledge [82.5462771088607]
  We propose a novel model selection metric specifically designed for ITE methods under the unsupervised domain adaptation setting. In particular, we propose selecting models whose predictions of interventions' effects satisfy known causal structures in the target domain.
  arXiv  Detail & Related papers  (2021-02-11T21:03:14Z)
• A Zero-Shot based Fingerprint Presentation Attack Detection System [8.676298469169174]
  We propose a novel Zero-Shot Presentation Attack Detection Model to guarantee the generalization of the PAD model. The proposed ZSPAD-Model based on generative model does not utilize any negative samples in the process of establishment. In order to improve the performance of the proposed model, 9 confidence scores are discussed in this article.
  arXiv  Detail & Related papers  (2020-02-12T10:52:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.