SoftED: Metrics for Soft Evaluation of Time Series Event Detection

• URL: http://arxiv.org/abs/2304.00439v2
• Date: Wed, 29 May 2024 08:56:49 GMT
• Title: SoftED: Metrics for Soft Evaluation of Time Series Event Detection
• Authors: Rebecca Salles, Janio Lima, Rafaelli Coutinho, Esther Pacitti, Florent Masseglia, Reza Akbarinia, Chao Chen, Jonathan Garibaldi, Fabio Porto, Eduardo Ogasawara,
• Abstract summary: Time series event detection methods are evaluated mainly by standard classification metrics that focus solely on detection accuracy. Inaccuracy in detecting an event can often result from its preceding or delayed effects reflected in neighboring detections. This paper introduces SoftED metrics, a new set of metrics designed for soft evaluating event detection methods.
• Score: 4.263111781491367
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Time series event detection methods are evaluated mainly by standard classification metrics that focus solely on detection accuracy. However, inaccuracy in detecting an event can often result from its preceding or delayed effects reflected in neighboring detections. These detections are valuable to trigger necessary actions or help mitigate unwelcome consequences. In this context, current metrics are insufficient and inadequate for the context of event detection. There is a demand for metrics that incorporate both the concept of time and temporal tolerance for neighboring detections. This paper introduces SoftED metrics, a new set of metrics designed for soft evaluating event detection methods. They enable the evaluation of both detection accuracy and the degree to which their detections represent events. They improved event detection evaluation by associating events and their representative detections, incorporating temporal tolerance in over 36\% of experiments compared to the usual classification metrics. SoftED metrics were validated by domain specialists that indicated their contribution to detection evaluation and method selection.

Related papers

• Event Detection via Probability Density Function Regression [0.0]
  This study introduces a generalized regression-based approach to reframe the time-interval-defined event detection problem. Inspired by heatmap regression techniques from computer vision, our approach aims to predict probability densities at event locations. We demonstrate that regression-based approaches outperform segmentation-based methods across various state-of-the-art baseline networks and datasets.
  arXiv  Detail & Related papers  (2024-08-23T01:58:56Z)
• Detection Latencies of Anomaly Detectors: An Overlooked Perspective ? [1.8492669447784602]
  In this paper, we argue the relevance of measuring the temporal latency of attacks and errors. We propose an evaluation approach for detectors to ensure a pragmatic trade-off between correct and in-time detection.
  arXiv  Detail & Related papers  (2024-02-14T10:52:39Z)
• On Pixel-level Performance Assessment in Anomaly Detection [87.7131059062292]
  Anomaly detection methods have demonstrated remarkable success across various applications. However, assessing their performance, particularly at the pixel-level, presents a complex challenge. In this paper, we dissect the intricacies of this challenge, underscored by visual evidence and statistical analysis.
  arXiv  Detail & Related papers  (2023-10-25T08:02:27Z)
• A Machine Learning-based Algorithm for Automated Detection of Frequency-based Events in Recorded Time Series of Sensor Data [0.0]
  This work proposes a novel event detection method that allows to identify frequency-based events in time series data. For the analysis of unseen time series data, events can be detected in their scalograms with the trained object detection model. The algorithm, evaluated on unseen datasets, achieves a precision rate of 0.97 in event detection.
  arXiv  Detail & Related papers  (2023-10-16T21:35:23Z)
• On the Universal Adversarial Perturbations for Efficient Data-free Adversarial Detection [55.73320979733527]
  We propose a data-agnostic adversarial detection framework, which induces different responses between normal and adversarial samples to UAPs. Experimental results show that our method achieves competitive detection performance on various text classification tasks.
  arXiv  Detail & Related papers  (2023-06-27T02:54:07Z)
• Beyond AUROC & co. for evaluating out-of-distribution detection performance [50.88341818412508]
  Given their relevance for safe(r) AI, it is important to examine whether the basis for comparing OOD detection methods is consistent with practical needs. We propose a new metric - Area Under the Threshold Curve (AUTC), which explicitly penalizes poor separation between ID and OOD samples.
  arXiv  Detail & Related papers  (2023-06-26T12:51:32Z)
• CertainNet: Sampling-free Uncertainty Estimation for Object Detection [65.28989536741658]
  Estimating the uncertainty of a neural network plays a fundamental role in safety-critical settings. In this work, we propose a novel sampling-free uncertainty estimation method for object detection. We call it CertainNet, and it is the first to provide separate uncertainties for each output signal: objectness, class, location and size.
  arXiv  Detail & Related papers  (2021-10-04T17:59:31Z)
• Bandit Quickest Changepoint Detection [55.855465482260165]
  Continuous monitoring of every sensor can be expensive due to resource constraints. We derive an information-theoretic lower bound on the detection delay for a general class of finitely parameterized probability distributions. We propose a computationally efficient online sensing scheme, which seamlessly balances the need for exploration of different sensing options with exploitation of querying informative actions.
  arXiv  Detail & Related papers  (2021-07-22T07:25:35Z)
• How Far Should We Look Back to Achieve Effective Real-Time Time-Series Anomaly Detection? [1.0437764544103274]
  Anomaly detection is the process of identifying unexpected events or ab-normalities in data. RePAD (Real-time Proactive Anomaly Detection algorithm) is a generic approach with all above-mentioned features. It is unclear how different amounts of historical data points affect the performance of RePAD.
  arXiv  Detail & Related papers  (2021-02-12T14:51:05Z)
• Statistical Evaluation of Anomaly Detectors for Sequences [0.0]
  We formalize a notion of precision and recall with temporal tolerance for point-based anomaly detection in sequential data. We show how to obtain null distributions for the two measures to assess the statistical significance of reported results.
  arXiv  Detail & Related papers  (2020-08-13T10:07:27Z)
• Learning to Evaluate Perception Models Using Planner-Centric Metrics [104.33349410009161]
  We propose a principled metric for 3D object detection specifically for the task of self-driving. We find that our metric penalizes many of the mistakes that other metrics penalize by design. For human evaluation, we generate scenes in which standard metrics and our metric disagree and find that humans side with our metric 79% of the time.
  arXiv  Detail & Related papers  (2020-04-19T02:14:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.