Autoencoder-based Anomaly Detection System for Online Data Quality Monitoring of the CMS Electromagnetic Calorimeter
- URL: http://arxiv.org/abs/2309.10157v2
- Date: Wed, 26 Jun 2024 12:45:55 GMT
- Title: Autoencoder-based Anomaly Detection System for Online Data Quality Monitoring of the CMS Electromagnetic Calorimeter
- Authors: The CMS ECAL Collaboration,
- Abstract summary: A real-time autoencoder-based anomaly detection system using semi-supervised machine learning is presented.
A novel method is introduced which maximizes the anomaly detection performance by exploiting the time-dependent evolution of anomalies.
The performance of the system is validated with anomalies found in 2018 and 2022 LHC collision data.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: The CMS detector is a general-purpose apparatus that detects high-energy collisions produced at the LHC. Online Data Quality Monitoring of the CMS electromagnetic calorimeter is a vital operational tool that allows detector experts to quickly identify, localize, and diagnose a broad range of detector issues that could affect the quality of physics data. A real-time autoencoder-based anomaly detection system using semi-supervised machine learning is presented enabling the detection of anomalies in the CMS electromagnetic calorimeter data. A novel method is introduced which maximizes the anomaly detection performance by exploiting the time-dependent evolution of anomalies as well as spatial variations in the detector response. The autoencoder-based system is able to efficiently detect anomalies, while maintaining a very low false discovery rate. The performance of the system is validated with anomalies found in 2018 and 2022 LHC collision data. Additionally, the first results from deploying the autoencoder-based system in the CMS online Data Quality Monitoring workflow during the beginning of Run 3 of the LHC are presented, showing its ability to detect issues missed by the existing system.
Related papers
- Explainable Online Unsupervised Anomaly Detection for Cyber-Physical Systems via Causal Discovery from Time Series [1.223779595809275]
State-of-the-art approaches based on deep learning via neural networks achieve outstanding performance at anomaly recognition.
We show that our method has higher training efficiency, outperforms the accuracy of state-of-the-art neural architectures.
arXiv Detail & Related papers (2024-04-15T15:42:12Z) - Spatio-Temporal Anomaly Detection with Graph Networks for Data Quality
Monitoring of the Hadron Calorimeter [1.1420116121141344]
The CMS experiment is a general-purpose detector for high-energy collision at the large hadron collider (HCL) at CERN.
It employs an online data quality monitoring (DQM) system to promptly spot and diagnose particle data acquisition problems to avoid data quality loss.
We propose the GraphSTAD system, which employs convolutional and graph neural networks to learn local spatial characteristics induced by particles traversing the detector.
arXiv Detail & Related papers (2023-11-07T18:33:08Z) - Autoencoder-based Online Data Quality Monitoring for the CMS
Electromagnetic Calorimeter [0.0]
A real-time autoencoder-based anomaly detection system is developed that is able to detect ECAL anomalies unseen in past data.
The new system is able to efficiently detect anomalies while maintaining an estimated false discovery rate between $10-2$ to $10-4$, beating existing benchmarks by about two orders of magnitude.
arXiv Detail & Related papers (2023-08-31T11:58:13Z) - Self-Supervised Masked Convolutional Transformer Block for Anomaly
Detection [122.4894940892536]
We present a novel self-supervised masked convolutional transformer block (SSMCTB) that comprises the reconstruction-based functionality at a core architectural level.
In this work, we extend our previous self-supervised predictive convolutional attentive block (SSPCAB) with a 3D masked convolutional layer, a transformer for channel-wise attention, as well as a novel self-supervised objective based on Huber loss.
arXiv Detail & Related papers (2022-09-25T04:56:10Z) - A Robust and Explainable Data-Driven Anomaly Detection Approach For
Power Electronics [56.86150790999639]
We present two anomaly detection and classification approaches, namely the Matrix Profile algorithm and anomaly transformer.
The Matrix Profile algorithm is shown to be well suited as a generalizable approach for detecting real-time anomalies in streaming time-series data.
A series of custom filters is created and added to the detector to tune its sensitivity, recall, and detection accuracy.
arXiv Detail & Related papers (2022-09-23T06:09:35Z) - A Temporal Anomaly Detection System for Vehicles utilizing Functional
Working Groups and Sensor Channels [0.0]
We introduce the Vehicle Performance, Reliability, and Operations dataset and use it to create a multi-phased approach to anomaly detection.
Our anomaly detection system can achieve 96% detection accuracy and accurately predicts 91% of true anomalies.
arXiv Detail & Related papers (2022-09-14T14:33:07Z) - Ranking-Based Physics-Informed Line Failure Detection in Power Grids [66.0797334582536]
Real-time and accurate detecting of potential line failures is the first step to mitigating the extreme weather impact and activating emergency controls.
Power balance equations nonlinearity, increased uncertainty in generation during extreme events, and lack of grid observability compromise the efficiency of traditional data-driven failure detection methods.
This paper proposes a Physics-InformEd Line failure Detector (FIELD) that leverages grid topology information to reduce sample and time complexities and improve localization accuracy.
arXiv Detail & Related papers (2022-08-31T18:19:25Z) - Smart Metering System Capable of Anomaly Detection by Bi-directional
LSTM Autoencoder [0.6649973446180738]
Anomaly detection is concerned with a wide range of applications such as fault detection, system monitoring, and event detection.
This paper presents an anomaly detection process to find outliers observed in the smart metering system.
BiLSTM based autoencoder is used and finds the anomalous data point.
arXiv Detail & Related papers (2021-12-06T12:34:59Z) - DAE : Discriminatory Auto-Encoder for multivariate time-series anomaly
detection in air transportation [68.8204255655161]
We propose a novel anomaly detection model called Discriminatory Auto-Encoder (DAE)
It uses the baseline of a regular LSTM-based auto-encoder but with several decoders, each getting data of a specific flight phase.
Results show that the DAE achieves better results in both accuracy and speed of detection.
arXiv Detail & Related papers (2021-09-08T14:07:55Z) - Anomaly Detection Based on Selection and Weighting in Latent Space [73.01328671569759]
We propose a novel selection-and-weighting-based anomaly detection framework called SWAD.
Experiments on both benchmark and real-world datasets have shown the effectiveness and superiority of SWAD.
arXiv Detail & Related papers (2021-03-08T10:56:38Z) - Adaptive Anomaly Detection for IoT Data in Hierarchical Edge Computing [71.86955275376604]
We propose an adaptive anomaly detection approach for hierarchical edge computing (HEC) systems to solve this problem.
We design an adaptive scheme to select one of the models based on the contextual information extracted from input data, to perform anomaly detection.
We evaluate our proposed approach using a real IoT dataset, and demonstrate that it reduces detection delay by 84% while maintaining almost the same accuracy as compared to offloading detection tasks to the cloud.
arXiv Detail & Related papers (2020-01-10T05:29:17Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.