Model Selection of Anomaly Detectors in the Absence of Labeled Validation Data

• URL: http://arxiv.org/abs/2310.10461v3
• Date: Mon, 16 Sep 2024 14:24:48 GMT
• Title: Model Selection of Anomaly Detectors in the Absence of Labeled Validation Data
• Authors: Clement Fung, Chen Qiu, Aodong Li, Maja Rudolph,
• Abstract summary: We propose SWSA: a framework to select image-based anomaly detectors without labeled validation data. Instead of collecting labeled validation data, we generate synthetic anomalies without any training or fine-tuning. Our synthetic anomalies are used to create detection tasks that compose a validation framework for model selection.
• Score: 18.233908098602114
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Anomaly detection is the task of identifying abnormal samples in large unlabeled datasets. While the advent of foundation models has produced powerful zero-shot anomaly detection methods, their deployment in practice is often hindered by the absence of labeled validation data -- without it, their detection performance cannot be evaluated reliably. In this work, we propose SWSA (Selection With Synthetic Anomalies): a general-purpose framework to select image-based anomaly detectors without labeled validation data. Instead of collecting labeled validation data, we generate synthetic anomalies without any training or fine-tuning, using only a small support set of normal images. Our synthetic anomalies are used to create detection tasks that compose a validation framework for model selection. In an empirical study, we evaluate SWSA with three types of synthetic anomalies and on two selection tasks: model selection of image-based anomaly detectors and prompt selection for CLIP-based anomaly detection. SWSA often selects models and prompts that match selections made with a ground-truth validation set, outperforming baseline selection strategies.

Related papers

• Towards Unsupervised Validation of Anomaly-Detection Models [5.439020425819001]
  This work presents a new paradigm to automated validation of anomaly-detection models, inspired by real-world, collaborative decision-making mechanisms. We focus on two commonly-used, unsupervised model-validation tasks -- model selection and model evaluation.
  arXiv  Detail & Related papers  (2024-10-18T16:27:04Z)
• Fine-grained Abnormality Prompt Learning for Zero-shot Anomaly Detection [88.34095233600719]
  FAPrompt is a novel framework designed to learn Fine-grained Abnormality Prompts for more accurate ZSAD. It substantially outperforms state-of-the-art methods by at least 3%-5% AUC/AP in both image- and pixel-level ZSAD tasks.
  arXiv  Detail & Related papers  (2024-10-14T08:41:31Z)
• ARC: A Generalist Graph Anomaly Detector with In-Context Learning [62.202323209244]
  ARC is a generalist GAD approach that enables a one-for-all'' GAD model to detect anomalies across various graph datasets on-the-fly. equipped with in-context learning, ARC can directly extract dataset-specific patterns from the target dataset. Extensive experiments on multiple benchmark datasets from various domains demonstrate the superior anomaly detection performance, efficiency, and generalizability of ARC.
  arXiv  Detail & Related papers  (2024-05-27T02:42:33Z)
• Generating and Reweighting Dense Contrastive Patterns for Unsupervised Anomaly Detection [59.34318192698142]
  We introduce a prior-less anomaly generation paradigm and develop an innovative unsupervised anomaly detection framework named GRAD. PatchDiff effectively expose various types of anomaly patterns. experiments on both MVTec AD and MVTec LOCO datasets also support the aforementioned observation.
  arXiv  Detail & Related papers  (2023-12-26T07:08:06Z)
• Open-Vocabulary Video Anomaly Detection [57.552523669351636]
  Video anomaly detection (VAD) with weak supervision has achieved remarkable performance in utilizing video-level labels to discriminate whether a video frame is normal or abnormal. Recent studies attempt to tackle a more realistic setting, open-set VAD, which aims to detect unseen anomalies given seen anomalies and normal videos. This paper takes a step further and explores open-vocabulary video anomaly detection (OVVAD), in which we aim to leverage pre-trained large models to detect and categorize seen and unseen anomalies.
  arXiv  Detail & Related papers  (2023-11-13T02:54:17Z)
• Active anomaly detection based on deep one-class classification [9.904380236739398]
  We tackle two essential problems of active learning for Deep SVDD: query strategy and semi-supervised learning method. First, rather than solely identifying anomalies, our query strategy selects uncertain samples according to an adaptive boundary. Second, we apply noise contrastive estimation in training a one-class classification model to incorporate both labeled normal and abnormal data effectively.
  arXiv  Detail & Related papers  (2023-09-18T03:56:45Z)
• SPADE: Semi-supervised Anomaly Detection under Distribution Mismatch [58.04518381476167]
  SPADE shows state-of-the-art semi-supervised anomaly detection performance across a wide range of scenarios with distribution mismatch. In some common real-world settings such as model facing new types of unlabeled anomalies, SPADE outperforms the state-of-the-art alternatives by 5% AUC in average.
  arXiv  Detail & Related papers  (2022-11-30T23:39:11Z)
• Unsupervised Model Selection for Time-series Anomaly Detection [7.8027110514393785]
  We identify three classes of surrogate (unsupervised) metrics, namely, prediction error, model centrality, and performance on injected synthetic anomalies. We formulate metric combination with multiple imperfect surrogate metrics as a robust rank aggregation problem. Large-scale experiments on multiple real-world datasets demonstrate that our proposed unsupervised approach is as effective as selecting the most accurate model.
  arXiv  Detail & Related papers  (2022-10-03T16:49:30Z)
• Self-Supervised Guided Segmentation Framework for Unsupervised Anomaly Detection [24.26958675342856]
  Unsupervised anomaly detection is a challenging task in industrial applications. The distribution gap between forged and real anomaly samples makes it difficult for models trained based on forged samples to effectively locate real anomalies. The Self-Supervised Guided Framework (SGSF) is proposed to generate forged anomalous samples and the normal sample features as the guidance information of segmentation for anomaly detection.
  arXiv  Detail & Related papers  (2022-09-26T06:14:56Z)
• Explainable Deep Few-shot Anomaly Detection with Deviation Networks [123.46611927225963]
  We introduce a novel weakly-supervised anomaly detection framework to train detection models. The proposed approach learns discriminative normality by leveraging the labeled anomalies and a prior probability. Our model is substantially more sample-efficient and robust, and performs significantly better than state-of-the-art competing methods in both closed-set and open-set settings.
  arXiv  Detail & Related papers  (2021-08-01T14:33:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.