Local Evaluation of Time Series Anomaly Detection Algorithms
- URL: http://arxiv.org/abs/2206.13167v1
- Date: Mon, 27 Jun 2022 10:18:41 GMT
- Title: Local Evaluation of Time Series Anomaly Detection Algorithms
- Authors: Alexis Huet and Jose Manuel Navarro and Dario Rossi
- Abstract summary: We show that an adversary algorithm can reach high precision and recall on almost any dataset under weak assumption.
We propose a theoretically grounded, robust, parameter-free and interpretable extension to precision/recall metrics.
- Score: 9.717823994163277
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In recent years, specific evaluation metrics for time series anomaly
detection algorithms have been developed to handle the limitations of the
classical precision and recall. However, such metrics are heuristically built
as an aggregate of multiple desirable aspects, introduce parameters and wipe
out the interpretability of the output. In this article, we first highlight the
limitations of the classical precision/recall, as well as the main issues of
the recent event-based metrics -- for instance, we show that an adversary
algorithm can reach high precision and recall on almost any dataset under weak
assumption. To cope with the above problems, we propose a theoretically
grounded, robust, parameter-free and interpretable extension to
precision/recall metrics, based on the concept of ``affiliation'' between the
ground truth and the prediction sets. Our metrics leverage measures of duration
between ground truth and predictions, and have thus an intuitive
interpretation. By further comparison against random sampling, we obtain a
normalized precision/recall, quantifying how much a given set of results is
better than a random baseline prediction. By construction, our approach keeps
the evaluation local regarding ground truth events, enabling fine-grained
visualization and interpretation of algorithmic results. We compare our
proposal against various public time series anomaly detection datasets,
algorithms and metrics. We further derive theoretical properties of the
affiliation metrics that give explicit expectations about their behavior and
ensure robustness against adversary strategies.
Related papers
- Likelihood Ratio Confidence Sets for Sequential Decision Making [51.66638486226482]
We revisit the likelihood-based inference principle and propose to use likelihood ratios to construct valid confidence sequences.
Our method is especially suitable for problems with well-specified likelihoods.
We show how to provably choose the best sequence of estimators and shed light on connections to online convex optimization.
arXiv Detail & Related papers (2023-11-08T00:10:21Z) - Score Matching-based Pseudolikelihood Estimation of Neural Marked
Spatio-Temporal Point Process with Uncertainty Quantification [59.81904428056924]
We introduce SMASH: a Score MAtching estimator for learning markedPs with uncertainty quantification.
Specifically, our framework adopts a normalization-free objective by estimating the pseudolikelihood of markedPs through score-matching.
The superior performance of our proposed framework is demonstrated through extensive experiments in both event prediction and uncertainty quantification.
arXiv Detail & Related papers (2023-10-25T02:37:51Z) - Leveraging Variational Autoencoders for Parameterized MMSE Estimation [10.141454378473972]
We propose a variational autoencoder-based framework for parameterizing a conditional linear minimum mean squared error estimator.
The derived estimator is shown to approximate the minimum mean squared error estimator by utilizing the variational autoencoder as a generative prior for the estimation problem.
We conduct a rigorous analysis by bounding the difference between the proposed and the minimum mean squared error estimator.
arXiv Detail & Related papers (2023-07-11T15:41:34Z) - Conformal Prediction with Missing Values [19.18178194789968]
We first show that the marginal coverage guarantee of conformal prediction holds on imputed data for any missingness distribution.
We then show that a universally consistent quantile regression algorithm trained on the imputed data is Bayes optimal for the pinball risk.
arXiv Detail & Related papers (2023-06-05T09:28:03Z) - Pedestrian Trajectory Forecasting Using Deep Ensembles Under Sensing
Uncertainty [125.41260574344933]
We consider an encoder-decoder based deep ensemble network for capturing both perception and predictive uncertainty simultaneously.
Overall, deep ensembles provided more robust predictions and the consideration of upstream uncertainty further increased the estimation accuracy for the model.
arXiv Detail & Related papers (2023-05-26T04:27:48Z) - Sequential Predictive Conformal Inference for Time Series [16.38369532102931]
We present a new distribution-free conformal prediction algorithm for sequential data (e.g., time series)
We specifically account for the nature that time series data are non-exchangeable, and thus many existing conformal prediction algorithms are not applicable.
arXiv Detail & Related papers (2022-12-07T05:07:27Z) - Deep Subspace Encoders for Nonlinear System Identification [0.0]
We propose a method which uses a truncated prediction loss and a subspace encoder for state estimation.
We show that, under mild conditions, the proposed method is locally consistent, increases optimization stability, and achieves increased data efficiency.
arXiv Detail & Related papers (2022-10-26T16:04:38Z) - Efficient and Differentiable Conformal Prediction with General Function
Classes [96.74055810115456]
We propose a generalization of conformal prediction to multiple learnable parameters.
We show that it achieves approximate valid population coverage and near-optimal efficiency within class.
Experiments show that our algorithm is able to learn valid prediction sets and improve the efficiency significantly.
arXiv Detail & Related papers (2022-02-22T18:37:23Z) - Dense Uncertainty Estimation [62.23555922631451]
In this paper, we investigate neural networks and uncertainty estimation techniques to achieve both accurate deterministic prediction and reliable uncertainty estimation.
We work on two types of uncertainty estimations solutions, namely ensemble based methods and generative model based methods, and explain their pros and cons while using them in fully/semi/weakly-supervised framework.
arXiv Detail & Related papers (2021-10-13T01:23:48Z) - Unifying Lower Bounds on Prediction Dimension of Consistent Convex
Surrogates [12.751555473216683]
Given a prediction task, understanding when one can and cannot design a consistent convex surrogate loss is an important area of machine learning research.
We unify these settings using tools from property elicitation, and give a general lower bound on prediction dimension.
Our lower bound tightens existing results in the case of discrete predictions, showing that previous calibration-based bounds can largely be recovered via property elicitation.
For continuous estimation, our lower bound resolves on open problem on estimating measures of risk and uncertainty.
arXiv Detail & Related papers (2021-02-16T15:29:05Z) - Orthogonal Statistical Learning [49.55515683387805]
We provide non-asymptotic excess risk guarantees for statistical learning in a setting where the population risk depends on an unknown nuisance parameter.
We show that if the population risk satisfies a condition called Neymanity, the impact of the nuisance estimation error on the excess risk bound achieved by the meta-algorithm is of second order.
arXiv Detail & Related papers (2019-01-25T02:21:24Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.