Position: Quo Vadis, Unsupervised Time Series Anomaly Detection?

• URL: http://arxiv.org/abs/2405.02678v3
• Date: Wed, 5 Jun 2024 13:12:17 GMT
• Title: Position: Quo Vadis, Unsupervised Time Series Anomaly Detection?
• Authors: M. Saquib Sarfraz, Mei-Yen Chen, Lukas Layer, Kunyu Peng, Marios Koulakis,
• Abstract summary: The current state of machine learning scholarship in Timeseries Anomaly Detection (TAD) is plagued by the persistent use of flawed evaluation metrics. Our paper presents a critical analysis of the status quo in TAD, revealing the misleading track of current research.
• Score: 11.269007806012931
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The current state of machine learning scholarship in Timeseries Anomaly Detection (TAD) is plagued by the persistent use of flawed evaluation metrics, inconsistent benchmarking practices, and a lack of proper justification for the choices made in novel deep learning-based model designs. Our paper presents a critical analysis of the status quo in TAD, revealing the misleading track of current research and highlighting problematic methods, and evaluation practices. Our position advocates for a shift in focus from solely pursuing novel model designs to improving benchmarking practices, creating non-trivial datasets, and critically evaluating the utility of complex methods against simpler baselines. Our findings demonstrate the need for rigorous evaluation protocols, the creation of simple baselines, and the revelation that state-of-the-art deep anomaly detection models effectively learn linear mappings. These findings suggest the need for more exploration and development of simple and interpretable TAD methods. The increment of model complexity in the state-of-the-art deep-learning based models unfortunately offers very little improvement. We offer insights and suggestions for the field to move forward. Code: https://github.com/ssarfraz/QuoVadisTAD

Related papers

• Effort: Efficient Orthogonal Modeling for Generalizable AI-Generated Image Detection [66.16595174895802]
  Existing AI-generated image (AIGI) detection methods often suffer from limited generalization performance. In this paper, we identify a crucial yet previously overlooked asymmetry phenomenon in AIGI detection.
  arXiv  Detail & Related papers  (2024-11-23T19:10:32Z)
• ActiveGLAE: A Benchmark for Deep Active Learning with Transformers [5.326702806697265]
  Deep active learning (DAL) seeks to reduce annotation costs by enabling the model to actively query instance annotations from which it expects to learn the most. There is currently no standardized evaluation protocol for transformer-based language models in the field of DAL. We propose the ActiveGLAE benchmark, a comprehensive collection of data sets and evaluation guidelines for assessing DAL.
  arXiv  Detail & Related papers  (2023-06-16T13:07:29Z)
• Universal Domain Adaptation from Foundation Models: A Baseline Study [58.51162198585434]
  We make empirical studies of state-of-the-art UniDA methods using foundation models. We introduce textitCLIP distillation, a parameter-free method specifically designed to distill target knowledge from CLIP models. Although simple, our method outperforms previous approaches in most benchmark tasks.
  arXiv  Detail & Related papers  (2023-05-18T16:28:29Z)
• Large-scale Pre-trained Models are Surprisingly Strong in Incremental Novel Class Discovery [76.63807209414789]
  We challenge the status quo in class-iNCD and propose a learning paradigm where class discovery occurs continuously and truly unsupervisedly. We propose simple baselines, composed of a frozen PTM backbone and a learnable linear classifier, that are not only simple to implement but also resilient under longer learning scenarios.
  arXiv  Detail & Related papers  (2023-03-28T13:47:16Z)
• Generalization Properties of Retrieval-based Models [50.35325326050263]
  Retrieval-based machine learning methods have enjoyed success on a wide range of problems. Despite growing literature showcasing the promise of these models, the theoretical underpinning for such models remains underexplored. We present a formal treatment of retrieval-based models to characterize their generalization ability.
  arXiv  Detail & Related papers  (2022-10-06T00:33:01Z)
• ALT-MAS: A Data-Efficient Framework for Active Testing of Machine Learning Algorithms [58.684954492439424]
  We propose a novel framework to efficiently test a machine learning model using only a small amount of labeled test data. The idea is to estimate the metrics of interest for a model-under-test using Bayesian neural network (BNN)
  arXiv  Detail & Related papers  (2021-04-11T12:14:04Z)
• Identifying Wrongly Predicted Samples: A Method for Active Learning [6.976600214375139]
  We propose a simple sample selection criterion that moves beyond uncertainty. We show state-of-the-art results and better rates at identifying wrongly predicted samples.
  arXiv  Detail & Related papers  (2020-10-14T09:00:42Z)
• Deep k-NN for Noisy Labels [55.97221021252733]
  We show that a simple $k$-nearest neighbor-based filtering approach on the logit layer of a preliminary model can remove mislabeled data and produce more accurate models than many recently proposed methods.
  arXiv  Detail & Related papers  (2020-04-26T05:15:36Z)
• Pattern Learning for Detecting Defect Reports and Improvement Requests in App Reviews [4.460358746823561]
  In this study, we follow novel approaches that target this absence of actionable insights by classifying reviews as defect reports and requests for improvement. We employ a supervised system that is capable of learning lexico-semantic patterns through genetic programming. We show that the automatically learned patterns outperform the manually created ones, to be generated.
  arXiv  Detail & Related papers  (2020-04-19T08:13:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.