Score-Based Change Detection for Gradient-Based Learning Machines

• URL: http://arxiv.org/abs/2106.14122v1
• Date: Sun, 27 Jun 2021 01:38:11 GMT
• Title: Score-Based Change Detection for Gradient-Based Learning Machines
• Authors: Lang Liu, Joseph Salmon, Zaid Harchaoui
• Abstract summary: We present a generic score-based change detection method that can detect a change in any number of components of a machine learning model trained via empirical risk minimization. We establish the consistency of the hypothesis test and show how to calibrate it to achieve a prescribed false alarm rate.
• Score: 9.670556223243182
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The widespread use of machine learning algorithms calls for automatic change detection algorithms to monitor their behavior over time. As a machine learning algorithm learns from a continuous, possibly evolving, stream of data, it is desirable and often critical to supplement it with a companion change detection algorithm to facilitate its monitoring and control. We present a generic score-based change detection method that can detect a change in any number of components of a machine learning model trained via empirical risk minimization. This proposed statistical hypothesis test can be readily implemented for such models designed within a differentiable programming framework. We establish the consistency of the hypothesis test and show how to calibrate it to achieve a prescribed false alarm rate. We illustrate the versatility of the approach on synthetic and real data.

Related papers

• Unsupervised Novelty Detection Methods Benchmarking with Wavelet Decomposition [0.22369578015657962]
  unsupervised machine learning algorithms for novelty detection are compared. A new dataset is gathered from an actuator vibrating at specific frequencies to benchmark the algorithms and evaluate the framework. Our findings offer valuable insights into the adaptability and robustness of unsupervised learning techniques for real-world novelty detection applications.
  arXiv  Detail & Related papers  (2024-09-11T09:31:28Z)
• Monitoring Machine Learning Models: Online Detection of Relevant Deviations [0.0]
  Machine learning models can degrade over time due to changes in data distribution or other factors. We propose a sequential monitoring scheme to detect relevant changes. Our research contributes a practical solution for distinguishing between minor fluctuations and meaningful degradations.
  arXiv  Detail & Related papers  (2023-09-26T18:46:37Z)
• Unsupervised Domain Adaptation for Self-Driving from Past Traversal Features [69.47588461101925]
  We propose a method to adapt 3D object detectors to new driving environments. Our approach enhances LiDAR-based detection models using spatial quantized historical features. Experiments on real-world datasets demonstrate significant improvements.
  arXiv  Detail & Related papers  (2023-09-21T15:00:31Z)
• Automatic Change-Point Detection in Time Series via Deep Learning [8.43086628139493]
  We show how to automatically generate new offline detection methods based on training a neural network. We present theory that quantifies the error rate for such an approach, and how it depends on the amount of training data. Our method also shows strong results in detecting and localising changes in activity based on accelerometer data.
  arXiv  Detail & Related papers  (2022-11-07T20:59:14Z)
• A Robust and Explainable Data-Driven Anomaly Detection Approach For Power Electronics [56.86150790999639]
  We present two anomaly detection and classification approaches, namely the Matrix Profile algorithm and anomaly transformer. The Matrix Profile algorithm is shown to be well suited as a generalizable approach for detecting real-time anomalies in streaming time-series data. A series of custom filters is created and added to the detector to tune its sensitivity, recall, and detection accuracy.
  arXiv  Detail & Related papers  (2022-09-23T06:09:35Z)
• Towards Diverse Evaluation of Class Incremental Learning: A Representation Learning Perspective [67.45111837188685]
  Class incremental learning (CIL) algorithms aim to continually learn new object classes from incrementally arriving data. We experimentally analyze neural network models trained by CIL algorithms using various evaluation protocols in representation learning.
  arXiv  Detail & Related papers  (2022-06-16T11:44:11Z)
• Learn then Test: Calibrating Predictive Algorithms to Achieve Risk Control [67.52000805944924]
  Learn then Test (LTT) is a framework for calibrating machine learning models. Our main insight is to reframe the risk-control problem as multiple hypothesis testing. We use our framework to provide new calibration methods for several core machine learning tasks with detailed worked examples in computer vision.
  arXiv  Detail & Related papers  (2021-10-03T17:42:03Z)
• A Novel Anomaly Detection Algorithm for Hybrid Production Systems based on Deep Learning and Timed Automata [73.38551379469533]
  DAD:DeepAnomalyDetection is a new approach for automatic model learning and anomaly detection in hybrid production systems. It combines deep learning and timed automata for creating behavioral model from observations. The algorithm has been applied to few data sets including two from real systems and has shown promising results.
  arXiv  Detail & Related papers  (2020-10-29T08:27:43Z)
• Sequential Changepoint Detection in Neural Networks with Checkpoints [11.763229353978321]
  We introduce a framework for online changepoint detection and simultaneous model learning. It is based on detecting changepoints across time by sequentially performing generalized likelihood ratio tests. We show improved performance compared to online Bayesian changepoint detection.
  arXiv  Detail & Related papers  (2020-10-06T21:49:54Z)
• Change Point Detection in Time Series Data using Autoencoders with a Time-Invariant Representation [69.34035527763916]
  Change point detection (CPD) aims to locate abrupt property changes in time series data. Recent CPD methods demonstrated the potential of using deep learning techniques, but often lack the ability to identify more subtle changes in the autocorrelation statistics of the signal. We employ an autoencoder-based methodology with a novel loss function, through which the used autoencoders learn a partially time-invariant representation that is tailored for CPD.
  arXiv  Detail & Related papers  (2020-08-21T15:03:21Z)
• Sequential Drift Detection in Deep Learning Classifiers [4.022057598291766]
  We utilize neural network embeddings to detect data drift by formulating the drift detection within an appropriate sequential decision framework. We introduce a loss function which evaluates an algorithm's ability to balance these two concerns.
  arXiv  Detail & Related papers  (2020-07-31T14:46:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.