Are Concept Drift Detectors Reliable Alarming Systems? -- A Comparative Study

• URL: http://arxiv.org/abs/2211.13098v1
• Date: Wed, 23 Nov 2022 16:31:15 GMT
• Title: Are Concept Drift Detectors Reliable Alarming Systems? -- A Comparative Study
• Authors: Lorena Poenaru-Olaru, Luis Cruz, Arie van Deursen, Jan S. Rellermeyer
• Abstract summary: Concept drift, also known as concept drift, impacts the performance of machine learning models. In this study, we assess the reliability of concept drift detectors to identify drift in time. Our findings aim to help practitioners understand which drift detector should be employed in different situations.
• Score: 6.7961908135481615
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As machine learning models increasingly replace traditional business logic in the production system, their lifecycle management is becoming a significant concern. Once deployed into production, the machine learning models are constantly evaluated on new streaming data. Given the continuous data flow, shifting data, also known as concept drift, is ubiquitous in such settings. Concept drift usually impacts the performance of machine learning models, thus, identifying the moment when concept drift occurs is required. Concept drift is identified through concept drift detectors. In this work, we assess the reliability of concept drift detectors to identify drift in time by exploring how late are they reporting drifts and how many false alarms are they signaling. We compare the performance of the most popular drift detectors belonging to two different concept drift detector groups, error rate-based detectors and data distribution-based detectors. We assess their performance on both synthetic and real-world data. In the case of synthetic data, we investigate the performance of detectors to identify two types of concept drift, abrupt and gradual. Our findings aim to help practitioners understand which drift detector should be employed in different situations and, to achieve this, we share a list of the most important observations made throughout this study, which can serve as guidelines for practical usage. Furthermore, based on our empirical results, we analyze the suitability of each concept drift detection group to be used as alarming system.

Related papers

• Learning 3D Perception from Others' Predictions [64.09115694891679]
  We investigate a new scenario to construct 3D object detectors: learning from the predictions of a nearby unit that is equipped with an accurate detector. For example, when a self-driving car enters a new area, it may learn from other traffic participants whose detectors have been optimized for that area.
  arXiv  Detail & Related papers  (2024-10-03T16:31:28Z)
• Unsupervised Concept Drift Detection from Deep Learning Representations in Real-time [5.999777817331315]
  Concept Drift is a phenomenon in which the underlying data distribution and statistical properties of a target domain change over time. We propose DriftLens, an unsupervised real-time concept drift detection framework. It works on unstructured data by exploiting the distribution distances of deep learning representations.
  arXiv  Detail & Related papers  (2024-06-24T23:41:46Z)
• Methods for Generating Drift in Text Streams [49.3179290313959]
  Concept drift is a frequent phenomenon in real-world datasets and corresponds to changes in data distribution over time. This paper provides four textual drift generation methods to ease the production of datasets with labeled drifts. Results show that all methods have their performance degraded right after the drifts, and the incremental SVM is the fastest to run and recover the previous performance levels.
  arXiv  Detail & Related papers  (2024-03-18T23:48:33Z)
• A comprehensive analysis of concept drift locality in data streams [3.5897534810405403]
  Concept drift must be detected for effective model adaptation to evolving data properties. We present a novel categorization of concept drift based on its locality and scale. We conduct a comparative assessment of 9 state-of-the-art drift detectors across diverse difficulties.
  arXiv  Detail & Related papers  (2023-11-10T20:57:43Z)
• Unsupervised Domain Adaptation for Self-Driving from Past Traversal Features [69.47588461101925]
  We propose a method to adapt 3D object detectors to new driving environments. Our approach enhances LiDAR-based detection models using spatial quantized historical features. Experiments on real-world datasets demonstrate significant improvements.
  arXiv  Detail & Related papers  (2023-09-21T15:00:31Z)
• Bayesian Autoencoders for Drift Detection in Industrial Environments [69.93875748095574]
  Autoencoders are unsupervised models which have been used for detecting anomalies in multi-sensor environments. Anomalies can come either from real changes in the environment (real drift) or from faulty sensory devices (virtual drift)
  arXiv  Detail & Related papers  (2021-07-28T10:19:58Z)
• Detecting Concept Drift With Neural Network Model Uncertainty [0.0]
  Uncertainty Drift Detection (UDD) is able to detect drifts without access to true labels. In contrast to input data-based drift detection, our approach considers the effects of the current input data on the properties of the prediction model. We show that UDD outperforms other state-of-the-art strategies on two synthetic as well as ten real-world data sets for both regression and classification tasks.
  arXiv  Detail & Related papers  (2021-07-05T08:56:36Z)
• Automatic Learning to Detect Concept Drift [40.69280758487987]
  We propose Meta-ADD, a novel framework that learns to classify concept drift by tracking the changed pattern of error rates. Specifically, in the training phase, we extract meta-features based on the error rates of various concept drift, after which a meta-detector is developed via prototypical neural network. In the detection phase, the learned meta-detector is fine-tuned to adapt to the corresponding data stream via stream-based active learning.
  arXiv  Detail & Related papers  (2021-05-04T11:10:39Z)
• A Novel Anomaly Detection Algorithm for Hybrid Production Systems based on Deep Learning and Timed Automata [73.38551379469533]
  DAD:DeepAnomalyDetection is a new approach for automatic model learning and anomaly detection in hybrid production systems. It combines deep learning and timed automata for creating behavioral model from observations. The algorithm has been applied to few data sets including two from real systems and has shown promising results.
  arXiv  Detail & Related papers  (2020-10-29T08:27:43Z)
• Adversarial Concept Drift Detection under Poisoning Attacks for Robust Data Stream Mining [15.49323098362628]
  We propose a framework for robust concept drift detection in the presence of adversarial and poisoning attacks. We introduce the taxonomy for two types of adversarial concept drifts, as well as a robust trainable drift detector. We also introduce Relative Loss of Robustness - a novel measure for evaluating the performance of concept drift detectors under poisoning attacks.
  arXiv  Detail & Related papers  (2020-09-20T18:46:31Z)
• Generalizable Pedestrian Detection: The Elephant In The Room [82.37430109152383]
  We find that existing state-of-the-art pedestrian detectors, though perform quite well when trained and tested on the same dataset, generalize poorly in cross dataset evaluation. We illustrate that diverse and dense datasets, collected by crawling the web, serve to be an efficient source of pre-training for pedestrian detection.
  arXiv  Detail & Related papers  (2020-03-19T14:14:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.