Sequential Drift Detection in Deep Learning Classifiers

• URL: http://arxiv.org/abs/2007.16109v1
• Date: Fri, 31 Jul 2020 14:46:21 GMT
• Title: Sequential Drift Detection in Deep Learning Classifiers
• Authors: Samuel Ackerman, Parijat Dube, Eitan Farchi
• Abstract summary: We utilize neural network embeddings to detect data drift by formulating the drift detection within an appropriate sequential decision framework. We introduce a loss function which evaluates an algorithm's ability to balance these two concerns.
• Score: 4.022057598291766
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We utilize neural network embeddings to detect data drift by formulating the drift detection within an appropriate sequential decision framework. This enables control of the false alarm rate although the statistical tests are repeatedly applied. Since change detection algorithms naturally face a tradeoff between avoiding false alarms and quick correct detection, we introduce a loss function which evaluates an algorithm's ability to balance these two concerns, and we use it in a series of experiments.

Related papers

• A Low-cost Strategic Monitoring Approach for Scalable and Interpretable Error Detection in Deep Neural Networks [6.537257913467249]
  We present a highly compact run-time monitoring approach for deep computer vision networks. It can efficiently detect silent data corruption originating from both hardware memory and input faults.
  arXiv  Detail & Related papers  (2023-10-31T10:45:55Z)
• PULL: Reactive Log Anomaly Detection Based On Iterative PU Learning [58.85063149619348]
  We propose PULL, an iterative log analysis method for reactive anomaly detection based on estimated failure time windows. Our evaluation shows that PULL consistently outperforms ten benchmark baselines across three different datasets.
  arXiv  Detail & Related papers  (2023-01-25T16:34:43Z)
• A Robust and Explainable Data-Driven Anomaly Detection Approach For Power Electronics [56.86150790999639]
  We present two anomaly detection and classification approaches, namely the Matrix Profile algorithm and anomaly transformer. The Matrix Profile algorithm is shown to be well suited as a generalizable approach for detecting real-time anomalies in streaming time-series data. A series of custom filters is created and added to the detector to tune its sensitivity, recall, and detection accuracy.
  arXiv  Detail & Related papers  (2022-09-23T06:09:35Z)
• From One to Many: A Deep Learning Coincident Gravitational-Wave Search [58.720142291102135]
  We construct a two-detector search for gravitational waves from binary black hole mergers using neural networks trained on non-spinning binary black hole data from a single detector. We find that none of these simple two-detector networks are capable of improving the sensitivity over applying networks individually to the data from the detectors.
  arXiv  Detail & Related papers  (2021-08-24T13:25:02Z)
• Out-of-Distribution Detection using Outlier Detection Methods [0.0]
  Out-of-distribution detection (OOD) deals with anomalous input to neural networks. We use outlier detection algorithms to detect anomalous input as reliable as specialized methods from the field of OOD. No neural network adaptation is required; detection is based on the model's softmax score.
  arXiv  Detail & Related papers  (2021-08-18T16:05:53Z)
• Bandit Quickest Changepoint Detection [55.855465482260165]
  Continuous monitoring of every sensor can be expensive due to resource constraints. We derive an information-theoretic lower bound on the detection delay for a general class of finitely parameterized probability distributions. We propose a computationally efficient online sensing scheme, which seamlessly balances the need for exploration of different sensing options with exploitation of querying informative actions.
  arXiv  Detail & Related papers  (2021-07-22T07:25:35Z)
• Detecting Concept Drift With Neural Network Model Uncertainty [0.0]
  Uncertainty Drift Detection (UDD) is able to detect drifts without access to true labels. In contrast to input data-based drift detection, our approach considers the effects of the current input data on the properties of the prediction model. We show that UDD outperforms other state-of-the-art strategies on two synthetic as well as ten real-world data sets for both regression and classification tasks.
  arXiv  Detail & Related papers  (2021-07-05T08:56:36Z)
• Score-Based Change Detection for Gradient-Based Learning Machines [9.670556223243182]
  We present a generic score-based change detection method that can detect a change in any number of components of a machine learning model trained via empirical risk minimization. We establish the consistency of the hypothesis test and show how to calibrate it to achieve a prescribed false alarm rate.
  arXiv  Detail & Related papers  (2021-06-27T01:38:11Z)
• Unsupervised Anomaly Detectors to Detect Intrusions in the Current Threat Landscape [0.11470070927586014]
  We show that Isolation Forests, One-Class Support Vector Machines and Self-Organizing Maps are more effective than their counterparts for intrusion detection. We detail how attacks with unstable, distributed or non-repeatable behavior as Fuzzing, Worms and Botnets are more difficult to detect.
  arXiv  Detail & Related papers  (2020-12-21T14:06:58Z)
• Change Point Detection in Time Series Data using Autoencoders with a Time-Invariant Representation [69.34035527763916]
  Change point detection (CPD) aims to locate abrupt property changes in time series data. Recent CPD methods demonstrated the potential of using deep learning techniques, but often lack the ability to identify more subtle changes in the autocorrelation statistics of the signal. We employ an autoencoder-based methodology with a novel loss function, through which the used autoencoders learn a partially time-invariant representation that is tailored for CPD.
  arXiv  Detail & Related papers  (2020-08-21T15:03:21Z)
• Bayesian Optimization with Machine Learning Algorithms Towards Anomaly Detection [66.05992706105224]
  In this paper, an effective anomaly detection framework is proposed utilizing Bayesian Optimization technique. The performance of the considered algorithms is evaluated using the ISCX 2012 dataset. Experimental results show the effectiveness of the proposed framework in term of accuracy rate, precision, low-false alarm rate, and recall.
  arXiv  Detail & Related papers  (2020-08-05T19:29:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.