Task-Sensitive Concept Drift Detector with Metric Learning

• URL: http://arxiv.org/abs/2108.06980v1
• Date: Mon, 16 Aug 2021 09:10:52 GMT
• Title: Task-Sensitive Concept Drift Detector with Metric Learning
• Authors: Andrea Castellani, Sebastian Schmitt, Barbara Hammer
• Abstract summary: We propose a novel task-sensitive drift detection framework, which is able to detect drifts without access to true labels during inference. It is able to detect real drift, where the drift affects the classification performance, while it properly ignores virtual drift. We evaluate the performance of the proposed framework with a novel metric, which accumulates the standard metrics of detection accuracy, false positive rate and detection delay into one value.
• Score: 7.706795195017394
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Detecting drifts in data is essential for machine learning applications, as changes in the statistics of processed data typically has a profound influence on the performance of trained models. Most of the available drift detection methods require access to true labels during inference time. In a real-world scenario, true labels usually available only during model training. In this work, we propose a novel task-sensitive drift detection framework, which is able to detect drifts without access to true labels during inference. It utilizes metric learning of a constrained low-dimensional embedding representation of the input data, which is best suited for the classification task. It is able to detect real drift, where the drift affects the classification performance, while it properly ignores virtual drift, where the classification performance is not affected by the drift. In the proposed framework, the actual method to detect a change in the statistics of incoming data samples can be chosen freely. We also propose the two change detection methods, which are based on the exponential moving average and a modified $z$-score, respectively. We evaluate the performance of the proposed framework with a novel metric, which accumulates the standard metrics of detection accuracy, false positive rate and detection delay into one value. Experimental evaluation on nine benchmarks datasets, with different types of drift, demonstrates that the proposed framework can reliably detect drifts, and outperforms state-of-the-art unsupervised drift detection approaches.

Related papers

• A Neighbor-Searching Discrepancy-based Drift Detection Scheme for Learning Evolving Data [40.00357483768265]
  This work presents a novel real concept drift detection method based on Neighbor-Searching Discrepancy. The proposed method is able to detect real concept drift with high accuracy while ignoring virtual drift. It can also indicate the direction of the classification boundary change by identifying the invasion or retreat of a certain class.
  arXiv  Detail & Related papers  (2024-05-23T04:03:36Z)
• Drift Detection: Introducing Gaussian Split Detector [1.9430846345184412]
  We introduce Gaussian Split Detector (GSD) a novel drift detector that works in batch mode. GSD is designed to work when the data follow a normal distribution and makes use of Gaussian mixture models to monitor changes in the decision boundary. We show that our detector outperforms the state of the art in detecting real drift and in ignoring virtual drift which is key to avoid false alarms.
  arXiv  Detail & Related papers  (2024-05-14T14:15:31Z)
• Unsupervised Domain Adaptation for Self-Driving from Past Traversal Features [69.47588461101925]
  We propose a method to adapt 3D object detectors to new driving environments. Our approach enhances LiDAR-based detection models using spatial quantized historical features. Experiments on real-world datasets demonstrate significant improvements.
  arXiv  Detail & Related papers  (2023-09-21T15:00:31Z)
• CADM: Confusion Model-based Detection Method for Real-drift in Chunk Data Stream [3.0885191226198785]
  Concept drift detection has attracted considerable attention due to its importance in many real-world applications such as health monitoring and fault diagnosis. We propose a new approach to detect real-drift in the chunk data stream with limited annotations based on concept confusion.
  arXiv  Detail & Related papers  (2023-03-25T08:59:27Z)
• Leveraging Unlabeled Data to Predict Out-of-Distribution Performance [63.740181251997306]
  Real-world machine learning deployments are characterized by mismatches between the source (training) and target (test) distributions. In this work, we investigate methods for predicting the target domain accuracy using only labeled source data and unlabeled target data. We propose Average Thresholded Confidence (ATC), a practical method that learns a threshold on the model's confidence, predicting accuracy as the fraction of unlabeled examples.
  arXiv  Detail & Related papers  (2022-01-11T23:01:12Z)
• Attentive Prototypes for Source-free Unsupervised Domain Adaptive 3D Object Detection [85.11649974840758]
  3D object detection networks tend to be biased towards the data they are trained on. We propose a single-frame approach for source-free, unsupervised domain adaptation of lidar-based 3D object detectors.
  arXiv  Detail & Related papers  (2021-11-30T18:42:42Z)
• Self-supervised Pretraining with Classification Labels for Temporal Activity Detection [54.366236719520565]
  Temporal Activity Detection aims to predict activity classes per frame. Due to the expensive frame-level annotations required for detection, the scale of detection datasets is limited. This work proposes a novel self-supervised pretraining method for detection leveraging classification labels.
  arXiv  Detail & Related papers  (2021-11-26T18:59:28Z)
• Bandit Quickest Changepoint Detection [55.855465482260165]
  Continuous monitoring of every sensor can be expensive due to resource constraints. We derive an information-theoretic lower bound on the detection delay for a general class of finitely parameterized probability distributions. We propose a computationally efficient online sensing scheme, which seamlessly balances the need for exploration of different sensing options with exploitation of querying informative actions.
  arXiv  Detail & Related papers  (2021-07-22T07:25:35Z)
• Detecting Concept Drift With Neural Network Model Uncertainty [0.0]
  Uncertainty Drift Detection (UDD) is able to detect drifts without access to true labels. In contrast to input data-based drift detection, our approach considers the effects of the current input data on the properties of the prediction model. We show that UDD outperforms other state-of-the-art strategies on two synthetic as well as ten real-world data sets for both regression and classification tasks.
  arXiv  Detail & Related papers  (2021-07-05T08:56:36Z)
• Concept Drift Detection via Equal Intensity k-means Space Partitioning [40.77597229122878]
  Cluster-based histogram called equal intensity k-means space partitioning (EI-kMeans) Three algorithms are developed to implement concept drift detection, including a greedy centroids algorithm, a cluster amplify-shrink algorithm, and a drift detection algorithm. Experiments on synthetic and real-world datasets demonstrate the advantages of EI-kMeans and show its efficacy in detecting concept drift.
  arXiv  Detail & Related papers  (2020-04-24T08:00:16Z)
• Learning with Out-of-Distribution Data for Audio Classification [60.48251022280506]
  We show that detecting and relabelling certain OOD instances, rather than discarding them, can have a positive effect on learning. The proposed method is shown to improve the performance of convolutional neural networks by a significant margin.
  arXiv  Detail & Related papers  (2020-02-11T21:08:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.