Revisiting OmniAnomaly for Anomaly Detection: performance metrics and comparison with PCA-based models

• URL: http://arxiv.org/abs/2603.18985v1
• Date: Thu, 19 Mar 2026 14:54:38 GMT
• Title: Revisiting OmniAnomaly for Anomaly Detection: performance metrics and comparison with PCA-based models
• Abstract summary: This work revisits OmniAnomaly, a widely used recurrent model for MTSAD, and compares it with a simple linear baseline based on Principal Component Analysis.<n>Both methods are evaluated under thresholding and evaluation procedures, with experiments repeated across 100 runs for each of the 28 machines in the dataset.<n>The results show large variability across machines and show that can achieve performance comparable to OmniAnomaly, and even outperform it when point-adjustment is not applied.
• Score: 0.08155575318208629
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep learning models have become the dominant approach for multivariate time series anomaly detection (MTSAD), often reporting substantial performance improvements over classical statistical methods. However, these gains are frequently evaluated under heterogeneous thresholding strategies and evaluation protocols, making fair comparisons difficult. This work revisits OmniAnomaly, a widely used stochastic recurrent model for MTSAD, and systematically compares it with a simple linear baseline based on Principal Component Analysis (PCA) on the Server Machine Dataset (SMD). Both methods are evaluated under identical thresholding and evaluation procedures, with experiments repeated across 100 runs for each of the 28 machines in the dataset. Performance is evaluated using Precision, Recall and F1-score at point-level, with and without point-adjustment, and under different aggregation strategies across machines and runs, with the corresponding standard deviations also reported. The results show large variability across machines and show that PCA can achieve performance comparable to OmniAnomaly, and even outperform it when point-adjustment is not applied. These findings question the added value of more complex architectures under current benchmarking practices and highlight the critical role of evaluation methodology in MTSAD research.

Related papers

• Enhancing Automated Machine Learning via Homogeneous Train-Test Splitting Methods [0.0]
  This paper investigates the role of statistical similarity in train-test splitting and its consequences for AutoML model evaluation.<n>Similarity is assessed using chi-square, Kolmogorov-Smirnov, and Maximum Mean Discrepancy (MMD) tests.<n>The proposed Optimised-Distribution method treats similarity as an explicit optimisation objective and achieves the highest mean MMD similarity, 89.0%, across all strategies evaluated.
  arXiv  Detail & Related papers  (2026-07-29T08:53:11Z)
• CollabEval: Statistically Efficient Collaborative Model Evaluation via Matrix Completion [65.53153291902454]
  Collaborative Evaluation (CollabEval) is a simple, effective, and principled method for exploiting dependencies between historical runs of different models on the same tasks to improve statistical efficiency.<n>We build a low-rank approximation of the score matrix, and use the reconstructed values to guarantee unbiased estimates of the true evaluation metric mean.<n>We find that CollabEval substantially reduces the mean confidence interval size, and the mean squared error of the point estimate, compared to baseline methods at the same annotation budget.
  arXiv  Detail & Related papers  (2026-07-06T13:22:04Z)
• Disjoint or Overlapping? Inference Windowing for Reconstruction-Based Time Series Anomaly Detection [1.5439079597766574]
  We study the role of the inference stride, which controls whether subsequences are processed as disjoint windows or with overlap.<n>We propose a unified training, tuning, and multi-seed evaluation protocol on the curated TSB-AD benchmark.<n>We show that overlapping windows yield consistent improvements, with average relative gain up to +28%, and can alter method rankings.
  arXiv  Detail & Related papers  (2026-06-02T19:42:30Z)
• Causal Effect Estimation with TMLE: Handling Missing Data and Near-Violations of Positivity [1.1546538067103258]
  We evaluate the performance of targeted maximum likelihood estimation (TMLE) for estimating the average treatment effect in missing data scenarios.<n>We employ model- and design-based simulations, with the latter using undersmoothed highly adaptive lasso on the 'WASH Benefits Bangladesh' dataset.<n>We compare eight missing data methods in conjunction with TMLE as the analysis method, distinguishing between non-multiple imputation (non-MI) and multiple imputation (MI) approaches.
  arXiv  Detail & Related papers  (2025-10-25T08:01:55Z)
• KG-EDAS: A Meta-Metric Framework for Evaluating Knowledge Graph Completion Models [0.0]
  A major challenge in evaluating Knowledge Graphs (KGs) is comparing their performance across multiple datasets and metrics.<n>We propose KG Evaluation based on Distance from Average Solution (EDAS) to integrate multi-metric, multi-dataset performance into a unified ranking.<n>EDAS offers a global perspective that supports more informed model selection and promotes fairness in cross-dataset evaluation.
  arXiv  Detail & Related papers  (2025-08-21T08:37:35Z)
• Generalization is not a universal guarantee: Estimating similarity to training data with an ensemble out-of-distribution metric [0.09363323206192666]
  Failure of machine learning models to generalize to new data is a core problem limiting the reliability of AI systems.<n>We propose a standardized approach for assessing data similarity by constructing a supervised autoencoder for generalizability estimation (SAGE)<n>We show that out-of-the-box model performance increases after SAGE score filtering, even when applied to data from the model's own training and test datasets.
  arXiv  Detail & Related papers  (2025-02-22T19:21:50Z)
• Revisiting BPR: A Replicability Study of a Common Recommender System Baseline [78.00363373925758]
  We study the features of the BPR model, indicating their impact on its performance, and investigate open-source BPR implementations. Our analysis reveals inconsistencies between these implementations and the original BPR paper, leading to a significant decrease in performance of up to 50% for specific implementations. We show that the BPR model can achieve performance levels close to state-of-the-art methods on the top-n recommendation tasks and even outperform them on specific datasets.
  arXiv  Detail & Related papers  (2024-09-21T18:39:53Z)
• Learning Feature Inversion for Multi-class Anomaly Detection under General-purpose COCO-AD Benchmark [101.23684938489413]
  Anomaly detection (AD) is often focused on detecting anomalies for industrial quality inspection and medical lesion examination. This work first constructs a large-scale and general-purpose COCO-AD dataset by extending COCO to the AD field. Inspired by the metrics in the segmentation field, we propose several more practical threshold-dependent AD-specific metrics.
  arXiv  Detail & Related papers  (2024-04-16T17:38:26Z)
• Consensus-Adaptive RANSAC [104.87576373187426]
  We propose a new RANSAC framework that learns to explore the parameter space by considering the residuals seen so far via a novel attention layer. The attention mechanism operates on a batch of point-to-model residuals, and updates a per-point estimation state to take into account the consensus found through a lightweight one-step transformer.
  arXiv  Detail & Related papers  (2023-07-26T08:25:46Z)
• GREAT Score: Global Robustness Evaluation of Adversarial Perturbation using Generative Models [60.48306899271866]
  We present a new framework, called GREAT Score, for global robustness evaluation of adversarial perturbation using generative models. We show high correlation and significantly reduced cost of GREAT Score when compared to the attack-based model ranking on RobustBench. GREAT Score can be used for remote auditing of privacy-sensitive black-box models.
  arXiv  Detail & Related papers  (2023-04-19T14:58:27Z)
• Doing Great at Estimating CATE? On the Neglected Assumptions in Benchmark Comparisons of Treatment Effect Estimators [91.3755431537592]
  We show that even in arguably the simplest setting, estimation under ignorability assumptions can be misleading. We consider two popular machine learning benchmark datasets for evaluation of heterogeneous treatment effect estimators. We highlight that the inherent characteristics of the benchmark datasets favor some algorithms over others.
  arXiv  Detail & Related papers  (2021-07-28T13:21:27Z)
• Model-based clustering of partial records [11.193504036335503]
  We develop clustering methodology through a model-based approach using the marginal density for the observed values. We compare our algorithm to the corresponding full expectation-maximization (EM) approach that considers the missing values in the incomplete data set. Simulation studies demonstrate that our approach has favorable recovery of the true cluster partition compared to case deletion and imputation.
  arXiv  Detail & Related papers  (2021-03-30T13:30:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.